Hydrogen-electricity coupling energy storage systems: Models, applications, and deep reinforcement learning algorithms
DOI:
https://doi.org/10.18686/cest.v2i1.96Keywords:
hydrogen storage; power systems; deep reinforcement learning; application scenariosAbstract
With the maturity of hydrogen storage technologies, hydrogen-electricity coupling energy storage in green electricity and green hydrogen modes is an ideal energy system. The construction of hydrogen-electricity coupling energy storage systems (HECESSs) is one of the important technological pathways for energy supply and deep decarbonization. In a HECESS, hydrogen storage can maintain the energy balance between supply and demand and increase the utilization efficiency of energy. However, its scenario models in power system establishment and the corresponding solution methods still need to be studied in depth. For accelerating the construction of HECESSs, firstly, this paper describes the current applications of hydrogen storage technologies from three aspects: hydrogen production, hydrogen power generation, and hydrogen storage. Secondly, based on the complementary synergistic mechanism of hydrogen energy and electric energy, the structure of the HECESS and its operation mode are described. To study the engineering applications of HECESSs more deeply, the recent progress of HECESS application at the source, grid, and load sides is reviewed. For the application of the models of hydrogen storage at the source/grid/load side, the selection of the solution method will affect the optimal solution of the model and solution efficiency. As solving complex multi-energy coupling models using traditional optimization methods is difficult, the paper therefore explored the advantages of deep reinforcement learning (DRL) algorithms and their applications in HECESSs. Finally, the technical application in the construction of new power systems supported by HECESSs is prospected. The study aims to provide a reference for the research on hydrogen storage in power systems.
References
Fuso Nerini F, Tomei J, To LS, et al. Mapping synergies and trade-offs between energy and the Sustainable Development Goals. Nature Energy. 2017, 3(1): 10-15. doi: 10.1038/s41560-017-0036-5
Iqbal M, Benmouna A, Becherif M, Mekhilef S. Survey on battery technologies and modeling methods for electric vehicles. Batteries. 2023; 9(3):185. doi: 10.3390/batteries9030185
Zhang Z, Kang C. Challenges and prospects for constructing the new-type power system towards a carbon neutrality future (Chinese). Proceedings of the CSEE. 2022; 42(8): 2806–2819.
Cai G, Kong L, Xue Y, Sun B. Overview of research on wind power coupled with hydrogen production technology. Automation of Electric Power Systems. 2014; 38(21): 127–135.
Pan G, Gu W, Zhang H, Qiu Y. Electricity and hydrogen energy system towards accomodation of high proportion of renewable energy. Automation of Electric Power Systems. 2020; 44(23): 1–10. doi: 10.7500/AEPS20200202003
Hanley ES, Deane J, Gallachóir BÓ. The role of hydrogen in low carbon energy futures–A review of existing perspectives. Renewable and Sustainable Energy Reviews. 2018, 82: 3027-3045. doi: 10.1016/j.rser.2017.10.034
Li J, Lin J, Xing X, et al. Technology portfolio selection and optimal planning of power-to-hydrogen (p2h) modules in active distribution network (Chinese). Proceedings of the CSEE. 2021; 41(12): 4021–4033. doi: 10.13334/j.0258-8013.pcsee.201307
Abe JO, Popoola API, Ajenifuja E, et al. Hydrogen energy, economy and storage: Review and recommendation. International Journal of Hydrogen Energy. 2019, 44(29): 15072-15086. doi: 10.1016/j.ijhydene.2019.04.068
Razzhivin IA, Suvorov AA, Ufa RA, et al. The energy storage mathematical models for simulation and comprehensive analysis of power system dynamics: A review. Part i. International Journal of Hydrogen Energy. 2023, 48(58): 22141-22160. doi: 10.1016/j.ijhydene.2023.03.070
Moradi R, Groth KM. Hydrogen storage and delivery: Review of the state of the art technologies and risk and reliability analysis. International Journal of Hydrogen Energy. 2019, 44(23): 12254-12269. doi: 10.1016/j.ijhydene.2019.03.041
Pei W, Zhang X, Deng W, et al. Review of operational control strategy for DC microgrids with electric-hydrogen hybrid storage systems. CSEE Journal of Power and Energy Systems. 2022; 8(2): 329–346. doi: 10.17775/CSEEJPES.2021.06960
Egeland-Eriksen T, Hajizadeh A, Sartori S. Hydrogen-based systems for integration of renewable energy in power systems: Achievements and perspectives. International Journal of Hydrogen Energy. 2021, 46(63): 31963-31983. doi: 10.1016/j.ijhydene.2021.06.218
Liu W, Sun L, Li Z, et al. Trends and future challenges in hydrogen production and storage research. Environmental Science and Pollution Research. 2020, 27(25): 31092-31104. doi: 10.1007/s11356-020-09470-0
Wang S, Kong L, Cai G, et al. Current status, challenges and prospects of key application technologies for hydrogen storage in power system (Chinese). Proceedings of the CSEE. 2023; 43(17): 6660–6681. doi: 10.13334/j.0258-8013.pcsee.230170
Yue M, Lambert H, Pahon E, et al. Hydrogen energy systems: A critical review of technologies, applications, trends and challenges. Renewable and Sustainable Energy Reviews. 2021, 146: 111180. doi: 10.1016/j.rser.2021.111180
Li Y, Wang D, Zhao F, Li F. Path, application and challenge of power-to-x (Chinese). Power System Technology.
; 1–14. doi: 10.13335/j.1000-3673.pst.2023.1234
Gao J, Song J, Wang JX, et al. Form and key technologies of integrated electricity-hydrogen system supporting energy security in China. Automation of Electric Power Systems. 2023; 47(19): 1–15.
Wu X, Li H, Wang X, et al. Cooperative Operation for Wind Turbines and Hydrogen Fueling Stations With On-Site Hydrogen Production. IEEE Transactions on Sustainable Energy. 2020, 11(4): 2775-2789. doi: 10.1109/tste.2020.2975609
Li Q, Zhao S, Pu Y, Chen W, Yu J. Capacity optimization of hybrid energy storage microgrid consid- ering electricity-hydrogen coupling. Transactions of China Electrotechnical Society 2021;36(3):486–95.
Xiong Y, Chen L, Zheng T, et al. Electricity-Heat-Hydrogen Modeling of Hydrogen Storage System Considering Off-Design Characteristics. IEEE Access. 2021, 9: 156768-156777. doi: 10.1109/access.2021.3130175
Liu L, Zhai R, Hu Y. Multi-objective optimization with advanced exergy analysis of a wind-solar‑hydrogen multi-energy supply system. Applied Energy. 2023, 348: 121512. doi: 10.1016/j.apenergy.2023.121512
Li J, Lin J, Song Y, et al. Operation Optimization of Power to Hydrogen and Heat (P2HH) in ADN Coordinated With the District Heating Network. IEEE Transactions on Sustainable Energy. 2019, 10(4): 1672-1683. doi: 10.1109/tste.2018.2868827
Pan G, Gu W, Lu Y, et al. Optimal Planning for Electricity-Hydrogen Integrated Energy System Considering Power to Hydrogen and Heat and Seasonal Storage. IEEE Transactions on Sustainable Energy. 2020, 11(4): 2662-2676. doi: 10.1109/tste.2020.2970078
Lin H, Wu Q, Chen X, et al. Economic and technological feasibility of using power-to-hydrogen technology under higher wind penetration in China. Renewable Energy. 2021, 173: 569-580. doi: 10.1016/j.renene.2021.04.015
Xiong J, Jiao Y, Wang M. A day-ahead optimal scheduling of regional integrated energy system considering power to gas. Modern Electric Power. 2022; 39(5): 554–561. doi: 10.19725/j.cnki.1007-2322.2021.0132
Tao Y, Qiu J, Lai S, et al. Collaborative Planning for Electricity Distribution Network and Transportation System Considering Hydrogen Fuel Cell Vehicles. IEEE Transactions on Transportation Electrification. 2020, 6(3): 1211-1225. doi: 10.1109/tte.2020.2996755
Li J, Lin J, Zhang H, et al. Optimal Investment of Electrolyzers and Seasonal Storages in Hydrogen Supply Chains Incorporated With Renewable Electric Networks. IEEE Transactions on Sustainable Energy. 2020, 11(3): 1773-1784. doi: 10.1109/tste.2019.2940604
El-Taweel NA, Khani H, Farag HEZ. Hydrogen Storage Optimal Scheduling for Fuel Supply and Capacity-Based Demand Response Program Under Dynamic Hydrogen Pricing. IEEE Transactions on Smart Grid. 2019, 10(4): 4531-4542. doi: 10.1109/tsg.2018.2863247
Pan G, Gu W, Qiu H, et al. Bi-level mixed-integer planning for electricity-hydrogen integrated energy system considering levelized cost of hydrogen. Applied Energy. 2020, 270: 115176. doi: 10.1016/j.apenergy.2020.115176
Zheng JH, Guo JC, Li Z, et al. Optimal design for a multi-level energy exploitation unit based on hydrogen storage combining methane reactor and carbon capture, utilization and storage. Journal of Energy Storage. 2023, 62: 106929. doi: 10.1016/j.est.2023.106929
Cheng Y, Zhang N, Lu Z, et al. Planning Multiple Energy Systems Toward Low-Carbon Society: A Decentralized Approach. IEEE Transactions on Smart Grid. 2019, 10(5): 4859-4869. doi: 10.1109/tsg.2018.2870323
Jiang Y, Yang G, Zhu Z. Wind-hydrogen-electricity coupled network planning considering traffic flow capture. Automation of Electric Power Systems. 2021; 45(22): 19–28.
Yang Y, Ma C, Lian C, et al. Optimal power reallocation of large-scale grid-connected photovoltaic power station integrated with hydrogen production. Journal of Cleaner Production. 2021, 298: 126830. doi: 10.1016/j.jclepro.2021.126830
Zhang Y, Hua QS, Sun L, et al. Life Cycle Optimization of Renewable Energy Systems Configuration with Hybrid Battery/Hydrogen Storage: A Comparative Study. Journal of Energy Storage. 2020, 30: 101470. doi: 10.1016/j.est.2020.101470
Garg H. A hybrid GSA-GA algorithm for constrained optimization problems. Information Sciences. 2019, 478: 499-523. doi: 10.1016/j.ins.2018.11.041
Xie Y, Sheng Y, Qiu M, et al. An adaptive decoding biased random key genetic algorithm for cloud workflow scheduling. Engineering Applications of Artificial Intelligence. 2022, 112: 104879. doi: 10.1016/j.engappai.2022.104879
Karimi-Mamaghan M, Mohammadi M, Meyer P, et al. Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: A state-of-the-art. European Journal of Operational Research. 2022, 296(2): 393-422. doi: 10.1016/j.ejor.2021.04.032
Sun H, Li Z, Chen A, et al. Current status and development trend of hydrogen production technology by wind power. Transactions of China Electrotechnical Society. 2019; 34(19): 4071–4083. doi: 10.19595/j.cnki.1000-6753.tces.180241
Peng S, Yang S, Yuan T. System dynamics of the evolutionary development of coupled wind-hydrogen-coal system for wind-coal enriched areas (Chinese). High Voltage Engineering. 2023; 49(8): 3478‒3489.
Fan H, Wang L, Xing M, et al. Coordinated scheduling of multiple buildings with electric-hydrogen complementary considering frequency stability constraints. Journal of Shanghai Jiaotong University. 2023; 57(12): 1559–1570. doi: 10.16183/j.cnki.jsjtu.2022.380
Xu S, Wang H, Cao Y, et al. Parameter tuning for self-synchronous voltage source doubly-fed wind turbines with stability boundary and multi-objective constraint. Automation of Electric Power Systems. 2023; 47(11): 18–28.
Wang F, Yang H, Li L, et al. Collaborative optimal method for electricity-hydrogen integrated energy system considering spatial-temporal characteristics of hydrogen transportation. Automation of Electric Power Systems. 2023; 47(19): 31–43.
Bai X, Fan Y, Liu Y, Song Y. Wind power storage virtual power plant considering reliability and flexibilitytiered capacity configuration. Power System Protection and Control. 2022; 50(8): 11–24.
Fan H, Yu W, Liu L, Dou Z. Multi-building coordinated dispatch in smart park for carbon emission peak and carbon neutrality considering electricity and hydrogen complementary. Automation of Electric Power Systems. 2022; 46(21): 42–51.
Ernst D, Glavic M, Capitanescu F, et al. Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 2009, 39(2): 517-529. doi: 10.1109/tsmcb.2008.2007630
Sharma P, Dutt Mathur H, Mishra P, et al. A critical and comparative review of energy management strategies for microgrids. Applied Energy. 2022, 327: 120028. doi: 10.1016/j.apenergy.2022.120028
Ceusters G, Rodríguez RC, García AB, et al. Model-predictive control and reinforcement learning in multi-energy system case studies. Applied Energy. 2021, 303: 117634. doi: 10.1016/j.apenergy.2021.117634
Sutton RS, Barto AG. Reinforcement Learning: An Introduction, 2nd ed. MIT Press; 2018.
Du G, Zou Y, Zhang X, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle. Energy. 2020, 201: 117591. doi: 10.1016/j.energy.2020.117591
Ding T, Zeng Z, Bai J, et al. Optimal Electric Vehicle Charging Strategy With Markov Decision Process and Reinforcement Learning Technique. IEEE Transactions on Industry Applications. 2020, 56(5): 5811-5823. doi: 10.1109/tia.2020.2990096
Li Y, Yu C, Shahidehpour M, et al. Deep Reinforcement Learning for Smart Grid Operations: Algorithms, Applications, and Prospects. Proceedings of the IEEE. 2023, 111(9): 1055-1096. doi: 10.1109/jproc.2023.3303358
Zheng JH, Wang WH, Li Z, Wu QH. Multi-layer double deep Q network for active distribution network equivalent modeling with internal identification for EV loads. Applied Soft Computing. 2023; 147: 110834. doi: 10.1016/j.asoc.2023.110834
Yang D, Wang L, Yu K, et al. A reinforcement learning-based energy management strategy for fuel cell hybrid vehicle considering real-time velocity prediction. Energy Conversion and Management. 2022, 274: 116453. doi: 10.1016/j.enconman.2022.116453
Wu JJ, Song DF, Zhang XM, et al. Multi-objective reinforcement learning-based energy management for fuel cell vehicles considering lifecycle costs. International Journal of Hydrogen Energy. 2023, 48(95): 37385-37401. doi: 10.1016/j.ijhydene.2023.06.145
Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
Ravanelli M, Brakel P, Omologo M, et al. Light Gated Recurrent Units for Speech Recognition. IEEE Transactions on Emerging Topics in Computational Intelligence. 2018, 2(2): 92-102. doi: 10.1109/tetci.2017.2762739
Bao S, Tang S, Sun P, et al. LSTM-based energy management algorithm for a vehicle power-split hybrid powertrain. Energy. 2023, 284: 129267. doi: 10.1016/j.energy.2023.129267
Zhang L, Zhang J, Gao T, et al. Improved LSTM based state of health estimation using random segments of the charging curves for lithium-ion batteries. Journal of Energy Storage. 2023, 74: 109370. doi: 10.1016/j.est.2023.109370
Jin R, Zhou Y, Lu C, et al. Deep reinforcement learning-based strategy for charging station participating in demand response. Applied Energy. 2022, 328: 120140. doi: 10.1016/j.apenergy.2022.120140
Lu S, Liu S, Zhu Y, et al. A DRL-Based Decentralized Computation Offloading Method: An Example of an Intelligent Manufacturing Scenario. IEEE Transactions on Industrial Informatics. 2023, 19(9): 9631-9641. doi: 10.1109/tii.2022.3227652
Hwang HS, Lee M, Seok J. Deep reinforcement learning with a critic-value-based branch tree for the inverse design of two-dimensional optical devices. Applied Soft Computing. 2022, 127: 109386. doi: 10.1016/j.asoc.2022.109386
Xiao H, Fu L, Shang C, et al. Ship energy scheduling with DQN-CE algorithm combining bi-directional LSTM and attention mechanism. Applied Energy. 2023, 347: 121378. doi: 10.1016/j.apenergy.2023.121378
Huang R, He H, Zhao X, et al. Longevity-aware energy management for fuel cell hybrid electric bus based on a novel proximal policy optimization deep reinforcement learning framework. Journal of Power Sources. 2023, 561: 232717. doi: 10.1016/j.jpowsour.2023.232717
Zeng S, Huang C, Wang F, et al. A Policy optimization-based Deep Reinforcement Learning method for data-driven output voltage control of grid connected solid oxide fuel cell considering operation constraints. Energy Reports. 2023, 10: 1161-1168. doi: 10.1016/j.egyr.2023.07.036
Martinez Cesena EA, Loukarakis E, Good N, et al. Integrated Electricity– Heat–Gas Systems: Techno–Economic Modeling, Optimization, and Application to Multienergy Districts. Proceedings of the IEEE. 2020, 108(9): 1392-1410. doi: 10.1109/jproc.2020.2989382
Yi Z, Luo Y, Westover T, et al. Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system. Applied Energy. 2022, 328: 120113. doi: 10.1016/j.apenergy.2022.120113
Yang T, Zhao L, Li W, et al. Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning. Energy. 2021, 235: 121377. doi: 10.1016/j.energy.2021.121377
Zhang G, Hu W, Cao D, et al. Data-driven optimal energy management for a wind-solar-diesel-battery-reverse osmosis hybrid energy system using a deep reinforcement learning approach. Energy Conversion and Management. 2021, 227: 113608. doi: 10.1016/j.enconman.2020.113608
Zhang B, Hu W, Cao D, et al. Deep reinforcement learning–based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energy Conversion and Management. 2019, 202: 112199. doi: 10.1016/j.enconman.2019.112199
Xu G, Lin Z, Wu Q, et al. Deep reinforcement learning based model-free optimization for unit commitment against wind power uncertainty. International Journal of Electrical Power & Energy Systems. 2024, 155: 109526. doi: 10.1016/j.ijepes.2023.109526
Alabi TM, Lawrence NP, Lu L, et al. Automated deep reinforcement learning for real-time scheduling strategy of multi-energy system integrated with post-carbon and direct-air carbon captured system. Applied Energy. 2023, 333: 120633. doi: 10.1016/j.apenergy.2022.120633
Monfaredi F, Shayeghi H, Siano P. Multi-agent deep reinforcement learning-based optimal energy management for grid-connected multiple energy carrier microgrids. International Journal of Electrical Power & Energy Systems. 2023, 153: 109292. doi: 10.1016/j.ijepes.2023.109292
Xiong K, Hu W, Cao D, et al. Coordinated energy management strategy for multi-energy hub with thermo-electrochemical effect based power-to-ammonia: A multi-agent deep reinforcement learning enabled approach. Renewable Energy. 2023, 214: 216-232. doi: 10.1016/j.renene.2023.05.067
Guo G, Zhang M, Gong Y, et al. Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay. Applied Energy. 2023, 349: 121648. doi: 10.1016/j.apenergy.2023.121648
Li Y, Bu F, Li Y, et al. Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach. Applied Energy. 2023, 333: 120540. doi: 10.1016/j.apenergy.2022.120540
Zhong S, Wang X, Zhao J, et al. Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating. Applied Energy. 2021, 288: 116623. doi: 10.1016/j.apenergy.2021.116623
Li J, Yu T, Zhang X. Coordinated load frequency control of multi-area integrated energy system using multi-agent deep reinforcement learning. Applied Energy. 2022, 306: 117900. doi: 10.1016/j.apenergy.2021.117900
Ye Y, Qiu D, Wu X, et al. Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning. IEEE Transactions on Smart Grid. 2020, 11(4): 3068-3082. doi: 10.1109/tsg.2020.2976771
Zhou S, Hu Z, Gu W, et al. Combined heat and power system intelligent economic dispatch: A deep reinforcement learning approach. International Journal of Electrical Power & Energy Systems. 2020, 120: 106016. doi: 10.1016/j.ijepes.2020.106016
Dong Y, Zhang H, Wang C, et al. Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy. Engineering Applications of Artificial Intelligence. 2024, 127: 107230. doi: 10.1016/j.engappai.2023.107230
Yun L, Wang D, Li L. Explainable multi-agent deep reinforcement learning for real-time demand response towards sustainable manufacturing. Applied Energy. 2023, 347: 121324. doi: 10.1016/j.apenergy.2023.121324
Xie J, Ajagekar A, You F. Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings. Applied Energy. 2023, 342: 121162. doi: 10.1016/j.apenergy.2023.121162
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Jiehui Zheng, Yingying Su, Wenhao Wang, Zhigang Li, Qinghua Wu
This work is licensed under a Creative Commons Attribution 4.0 International License.
