Hydrogen-electricity coupling energy storage systems: Models, applications, and deep reinforcement learning algorithms
DOI:
https://doi.org/10.18686/cest.v2i1.96Keywords:
hydrogen storage; power systems; deep reinforcement learning; application scenariosAbstract
With the maturity of hydrogen storage technologies, hydrogen-electricity coupling energy storage in green electricity and green hydrogen modes is an ideal energy system. The construction of hydrogen-electricity coupling energy storage systems (HECESSs) is one of the important technological pathways for energy supply and deep decarbonization. In a HECESS, hydrogen storage can maintain the energy balance between supply and demand and increase the utilization efficiency of energy. However, its scenario models in power system establishment and the corresponding solution methods still need to be studied in depth. For accelerating the construction of HECESSs, firstly, this paper describes the current applications of hydrogen storage technologies from three aspects: hydrogen production, hydrogen power generation, and hydrogen storage. Secondly, based on the complementary synergistic mechanism of hydrogen energy and electric energy, the structure of the HECESS and its operation mode are described. To study the engineering applications of HECESSs more deeply, the recent progress of HECESS application at the source, grid, and load sides is reviewed. For the application of the models of hydrogen storage at the source/grid/load side, the selection of the solution method will affect the optimal solution of the model and solution efficiency. As solving complex multi-energy coupling models using traditional optimization methods is difficult, the paper therefore explored the advantages of deep reinforcement learning (DRL) algorithms and their applications in HECESSs. Finally, the technical application in the construction of new power systems supported by HECESSs is prospected. The study aims to provide a reference for the research on hydrogen storage in power systems.
References
1. Fuso Nerini F, Tomei J, To LS, et al. Mapping synergies and trade-offs between energy and the Sustainable Development Goals. Nature Energy. 2017, 3(1): 10-15. doi: 10.1038/s41560-017-0036-5
2. Iqbal M, Benmouna A, Becherif M, Mekhilef S. Survey on battery technologies and modeling methods for electric vehicles. Batteries. 2023; 9(3):185. doi: 10.3390/batteries9030185
3. Zhang Z, Kang C. Challenges and prospects for constructing the new-type power system towards a carbon neutrality future (Chinese). Proceedings of the CSEE. 2022; 42(8): 2806–2819.
4. Cai G, Kong L, Xue Y, Sun B. Overview of research on wind power coupled with hydrogen production technology. Automation of Electric Power Systems. 2014; 38(21): 127–135.
5. Pan G, Gu W, Zhang H, Qiu Y. Electricity and hydrogen energy system towards accomodation of high proportion of renewable energy. Automation of Electric Power Systems. 2020; 44(23): 1–10. doi: 10.7500/AEPS20200202003
6. Hanley ES, Deane J, Gallachóir BÓ. The role of hydrogen in low carbon energy futures–A review of existing perspectives. Renewable and Sustainable Energy Reviews. 2018, 82: 3027-3045. doi: 10.1016/j.rser.2017.10.034
7. Li J, Lin J, Xing X, et al. Technology portfolio selection and optimal planning of power-to-hydrogen (p2h) modules in active distribution network (Chinese). Proceedings of the CSEE. 2021; 41(12): 4021–4033. doi: 10.13334/j.0258-8013.pcsee.201307
8. Abe JO, Popoola API, Ajenifuja E, et al. Hydrogen energy, economy and storage: Review and recommendation. International Journal of Hydrogen Energy. 2019, 44(29): 15072-15086. doi: 10.1016/j.ijhydene.2019.04.068
9. Razzhivin IA, Suvorov AA, Ufa RA, et al. The energy storage mathematical models for simulation and comprehensive analysis of power system dynamics: A review. Part i. International Journal of Hydrogen Energy. 2023, 48(58): 22141-22160. doi: 10.1016/j.ijhydene.2023.03.070
10. Moradi R, Groth KM. Hydrogen storage and delivery: Review of the state of the art technologies and risk and reliability analysis. International Journal of Hydrogen Energy. 2019, 44(23): 12254-12269. doi: 10.1016/j.ijhydene.2019.03.041
11. Pei W, Zhang X, Deng W, et al. Review of operational control strategy for DC microgrids with electric-hydrogen hybrid storage systems. CSEE Journal of Power and Energy Systems. 2022; 8(2): 329–346. doi: 10.17775/CSEEJPES.2021.06960
12. Egeland-Eriksen T, Hajizadeh A, Sartori S. Hydrogen-based systems for integration of renewable energy in power systems: Achievements and perspectives. International Journal of Hydrogen Energy. 2021, 46(63): 31963-31983. doi: 10.1016/j.ijhydene.2021.06.218
13. Liu W, Sun L, Li Z, et al. Trends and future challenges in hydrogen production and storage research. Environmental Science and Pollution Research. 2020, 27(25): 31092-31104. doi: 10.1007/s11356-020-09470-0
14. Wang S, Kong L, Cai G, et al. Current status, challenges and prospects of key application technologies for hydrogen storage in power system (Chinese). Proceedings of the CSEE. 2023; 43(17): 6660–6681. doi: 10.13334/j.0258-8013.pcsee.230170
15. Yue M, Lambert H, Pahon E, et al. Hydrogen energy systems: A critical review of technologies, applications, trends and challenges. Renewable and Sustainable Energy Reviews. 2021, 146: 111180. doi: 10.1016/j.rser.2021.111180
16. Li Y, Wang D, Zhao F, Li F. Path, application and challenge of power-to-x (Chinese). Power System Technology.
17. ; 1–14. doi: 10.13335/j.1000-3673.pst.2023.1234
18. Gao J, Song J, Wang JX, et al. Form and key technologies of integrated electricity-hydrogen system supporting energy security in China. Automation of Electric Power Systems. 2023; 47(19): 1–15.
19. Wu X, Li H, Wang X, et al. Cooperative Operation for Wind Turbines and Hydrogen Fueling Stations With On-Site Hydrogen Production. IEEE Transactions on Sustainable Energy. 2020, 11(4): 2775-2789. doi: 10.1109/tste.2020.2975609
20. Li Q, Zhao S, Pu Y, Chen W, Yu J. Capacity optimization of hybrid energy storage microgrid consid- ering electricity-hydrogen coupling. Transactions of China Electrotechnical Society 2021;36(3):486–95.
21. Xiong Y, Chen L, Zheng T, et al. Electricity-Heat-Hydrogen Modeling of Hydrogen Storage System Considering Off-Design Characteristics. IEEE Access. 2021, 9: 156768-156777. doi: 10.1109/access.2021.3130175
22. Liu L, Zhai R, Hu Y. Multi-objective optimization with advanced exergy analysis of a wind-solar‑hydrogen multi-energy supply system. Applied Energy. 2023, 348: 121512. doi: 10.1016/j.apenergy.2023.121512
23. Li J, Lin J, Song Y, et al. Operation Optimization of Power to Hydrogen and Heat (P2HH) in ADN Coordinated With the District Heating Network. IEEE Transactions on Sustainable Energy. 2019, 10(4): 1672-1683. doi: 10.1109/tste.2018.2868827
24. Pan G, Gu W, Lu Y, et al. Optimal Planning for Electricity-Hydrogen Integrated Energy System Considering Power to Hydrogen and Heat and Seasonal Storage. IEEE Transactions on Sustainable Energy. 2020, 11(4): 2662-2676. doi: 10.1109/tste.2020.2970078
25. Lin H, Wu Q, Chen X, et al. Economic and technological feasibility of using power-to-hydrogen technology under higher wind penetration in China. Renewable Energy. 2021, 173: 569-580. doi: 10.1016/j.renene.2021.04.015
26. Xiong J, Jiao Y, Wang M. A day-ahead optimal scheduling of regional integrated energy system considering power to gas. Modern Electric Power. 2022; 39(5): 554–561. doi: 10.19725/j.cnki.1007-2322.2021.0132
27. Tao Y, Qiu J, Lai S, et al. Collaborative Planning for Electricity Distribution Network and Transportation System Considering Hydrogen Fuel Cell Vehicles. IEEE Transactions on Transportation Electrification. 2020, 6(3): 1211-1225. doi: 10.1109/tte.2020.2996755
28. Li J, Lin J, Zhang H, et al. Optimal Investment of Electrolyzers and Seasonal Storages in Hydrogen Supply Chains Incorporated With Renewable Electric Networks. IEEE Transactions on Sustainable Energy. 2020, 11(3): 1773-1784. doi: 10.1109/tste.2019.2940604
29. El-Taweel NA, Khani H, Farag HEZ. Hydrogen Storage Optimal Scheduling for Fuel Supply and Capacity-Based Demand Response Program Under Dynamic Hydrogen Pricing. IEEE Transactions on Smart Grid. 2019, 10(4): 4531-4542. doi: 10.1109/tsg.2018.2863247
30. Pan G, Gu W, Qiu H, et al. Bi-level mixed-integer planning for electricity-hydrogen integrated energy system considering levelized cost of hydrogen. Applied Energy. 2020, 270: 115176. doi: 10.1016/j.apenergy.2020.115176
31. Zheng JH, Guo JC, Li Z, et al. Optimal design for a multi-level energy exploitation unit based on hydrogen storage combining methane reactor and carbon capture, utilization and storage. Journal of Energy Storage. 2023, 62: 106929. doi: 10.1016/j.est.2023.106929
32. Cheng Y, Zhang N, Lu Z, et al. Planning Multiple Energy Systems Toward Low-Carbon Society: A Decentralized Approach. IEEE Transactions on Smart Grid. 2019, 10(5): 4859-4869. doi: 10.1109/tsg.2018.2870323
33. Jiang Y, Yang G, Zhu Z. Wind-hydrogen-electricity coupled network planning considering traffic flow capture. Automation of Electric Power Systems. 2021; 45(22): 19–28.
34. Yang Y, Ma C, Lian C, et al. Optimal power reallocation of large-scale grid-connected photovoltaic power station integrated with hydrogen production. Journal of Cleaner Production. 2021, 298: 126830. doi: 10.1016/j.jclepro.2021.126830
35. Zhang Y, Hua QS, Sun L, et al. Life Cycle Optimization of Renewable Energy Systems Configuration with Hybrid Battery/Hydrogen Storage: A Comparative Study. Journal of Energy Storage. 2020, 30: 101470. doi: 10.1016/j.est.2020.101470
36. Garg H. A hybrid GSA-GA algorithm for constrained optimization problems. Information Sciences. 2019, 478: 499-523. doi: 10.1016/j.ins.2018.11.041
37. Xie Y, Sheng Y, Qiu M, et al. An adaptive decoding biased random key genetic algorithm for cloud workflow scheduling. Engineering Applications of Artificial Intelligence. 2022, 112: 104879. doi: 10.1016/j.engappai.2022.104879
38. Karimi-Mamaghan M, Mohammadi M, Meyer P, et al. Machine learning at the service of meta-heuristics for solving combinatorial optimization problems: A state-of-the-art. European Journal of Operational Research. 2022, 296(2): 393-422. doi: 10.1016/j.ejor.2021.04.032
39. Sun H, Li Z, Chen A, et al. Current status and development trend of hydrogen production technology by wind power. Transactions of China Electrotechnical Society. 2019; 34(19): 4071–4083. doi: 10.19595/j.cnki.1000-6753.tces.180241
40. Peng S, Yang S, Yuan T. System dynamics of the evolutionary development of coupled wind-hydrogen-coal system for wind-coal enriched areas (Chinese). High Voltage Engineering. 2023; 49(8): 3478‒3489.
41. Fan H, Wang L, Xing M, et al. Coordinated scheduling of multiple buildings with electric-hydrogen complementary considering frequency stability constraints. Journal of Shanghai Jiaotong University. 2023; 57(12): 1559–1570. doi: 10.16183/j.cnki.jsjtu.2022.380
42. Xu S, Wang H, Cao Y, et al. Parameter tuning for self-synchronous voltage source doubly-fed wind turbines with stability boundary and multi-objective constraint. Automation of Electric Power Systems. 2023; 47(11): 18–28.
43. Wang F, Yang H, Li L, et al. Collaborative optimal method for electricity-hydrogen integrated energy system considering spatial-temporal characteristics of hydrogen transportation. Automation of Electric Power Systems. 2023; 47(19): 31–43.
44. Bai X, Fan Y, Liu Y, Song Y. Wind power storage virtual power plant considering reliability and flexibilitytiered capacity configuration. Power System Protection and Control. 2022; 50(8): 11–24.
45. Fan H, Yu W, Liu L, Dou Z. Multi-building coordinated dispatch in smart park for carbon emission peak and carbon neutrality considering electricity and hydrogen complementary. Automation of Electric Power Systems. 2022; 46(21): 42–51.
46. Ernst D, Glavic M, Capitanescu F, et al. Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics). 2009, 39(2): 517-529. doi: 10.1109/tsmcb.2008.2007630
47. Sharma P, Dutt Mathur H, Mishra P, et al. A critical and comparative review of energy management strategies for microgrids. Applied Energy. 2022, 327: 120028. doi: 10.1016/j.apenergy.2022.120028
48. Ceusters G, Rodríguez RC, García AB, et al. Model-predictive control and reinforcement learning in multi-energy system case studies. Applied Energy. 2021, 303: 117634. doi: 10.1016/j.apenergy.2021.117634
49. Sutton RS, Barto AG. Reinforcement Learning: An Introduction, 2nd ed. MIT Press; 2018.
50. Du G, Zou Y, Zhang X, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle. Energy. 2020, 201: 117591. doi: 10.1016/j.energy.2020.117591
51. Ding T, Zeng Z, Bai J, et al. Optimal Electric Vehicle Charging Strategy With Markov Decision Process and Reinforcement Learning Technique. IEEE Transactions on Industry Applications. 2020, 56(5): 5811-5823. doi: 10.1109/tia.2020.2990096
52. Li Y, Yu C, Shahidehpour M, et al. Deep Reinforcement Learning for Smart Grid Operations: Algorithms, Applications, and Prospects. Proceedings of the IEEE. 2023, 111(9): 1055-1096. doi: 10.1109/jproc.2023.3303358
53. Zheng JH, Wang WH, Li Z, Wu QH. Multi-layer double deep Q network for active distribution network equivalent modeling with internal identification for EV loads. Applied Soft Computing. 2023; 147: 110834. doi: 10.1016/j.asoc.2023.110834
54. Yang D, Wang L, Yu K, et al. A reinforcement learning-based energy management strategy for fuel cell hybrid vehicle considering real-time velocity prediction. Energy Conversion and Management. 2022, 274: 116453. doi: 10.1016/j.enconman.2022.116453
55. Wu JJ, Song DF, Zhang XM, et al. Multi-objective reinforcement learning-based energy management for fuel cell vehicles considering lifecycle costs. International Journal of Hydrogen Energy. 2023, 48(95): 37385-37401. doi: 10.1016/j.ijhydene.2023.06.145
56. Lecun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE. 1998, 86(11): 2278-2324. doi: 10.1109/5.726791
57. Ravanelli M, Brakel P, Omologo M, et al. Light Gated Recurrent Units for Speech Recognition. IEEE Transactions on Emerging Topics in Computational Intelligence. 2018, 2(2): 92-102. doi: 10.1109/tetci.2017.2762739
58. Bao S, Tang S, Sun P, et al. LSTM-based energy management algorithm for a vehicle power-split hybrid powertrain. Energy. 2023, 284: 129267. doi: 10.1016/j.energy.2023.129267
59. Zhang L, Zhang J, Gao T, et al. Improved LSTM based state of health estimation using random segments of the charging curves for lithium-ion batteries. Journal of Energy Storage. 2023, 74: 109370. doi: 10.1016/j.est.2023.109370
60. Jin R, Zhou Y, Lu C, et al. Deep reinforcement learning-based strategy for charging station participating in demand response. Applied Energy. 2022, 328: 120140. doi: 10.1016/j.apenergy.2022.120140
61. Lu S, Liu S, Zhu Y, et al. A DRL-Based Decentralized Computation Offloading Method: An Example of an Intelligent Manufacturing Scenario. IEEE Transactions on Industrial Informatics. 2023, 19(9): 9631-9641. doi: 10.1109/tii.2022.3227652
62. Hwang HS, Lee M, Seok J. Deep reinforcement learning with a critic-value-based branch tree for the inverse design of two-dimensional optical devices. Applied Soft Computing. 2022, 127: 109386. doi: 10.1016/j.asoc.2022.109386
63. Xiao H, Fu L, Shang C, et al. Ship energy scheduling with DQN-CE algorithm combining bi-directional LSTM and attention mechanism. Applied Energy. 2023, 347: 121378. doi: 10.1016/j.apenergy.2023.121378
64. Huang R, He H, Zhao X, et al. Longevity-aware energy management for fuel cell hybrid electric bus based on a novel proximal policy optimization deep reinforcement learning framework. Journal of Power Sources. 2023, 561: 232717. doi: 10.1016/j.jpowsour.2023.232717
65. Zeng S, Huang C, Wang F, et al. A Policy optimization-based Deep Reinforcement Learning method for data-driven output voltage control of grid connected solid oxide fuel cell considering operation constraints. Energy Reports. 2023, 10: 1161-1168. doi: 10.1016/j.egyr.2023.07.036
66. Martinez Cesena EA, Loukarakis E, Good N, et al. Integrated Electricity– Heat–Gas Systems: Techno–Economic Modeling, Optimization, and Application to Multienergy Districts. Proceedings of the IEEE. 2020, 108(9): 1392-1410. doi: 10.1109/jproc.2020.2989382
67. Yi Z, Luo Y, Westover T, et al. Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system. Applied Energy. 2022, 328: 120113. doi: 10.1016/j.apenergy.2022.120113
68. Yang T, Zhao L, Li W, et al. Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning. Energy. 2021, 235: 121377. doi: 10.1016/j.energy.2021.121377
69. Zhang G, Hu W, Cao D, et al. Data-driven optimal energy management for a wind-solar-diesel-battery-reverse osmosis hybrid energy system using a deep reinforcement learning approach. Energy Conversion and Management. 2021, 227: 113608. doi: 10.1016/j.enconman.2020.113608
70. Zhang B, Hu W, Cao D, et al. Deep reinforcement learning–based approach for optimizing energy conversion in integrated electrical and heating system with renewable energy. Energy Conversion and Management. 2019, 202: 112199. doi: 10.1016/j.enconman.2019.112199
71. Xu G, Lin Z, Wu Q, et al. Deep reinforcement learning based model-free optimization for unit commitment against wind power uncertainty. International Journal of Electrical Power & Energy Systems. 2024, 155: 109526. doi: 10.1016/j.ijepes.2023.109526
72. Alabi TM, Lawrence NP, Lu L, et al. Automated deep reinforcement learning for real-time scheduling strategy of multi-energy system integrated with post-carbon and direct-air carbon captured system. Applied Energy. 2023, 333: 120633. doi: 10.1016/j.apenergy.2022.120633
73. Monfaredi F, Shayeghi H, Siano P. Multi-agent deep reinforcement learning-based optimal energy management for grid-connected multiple energy carrier microgrids. International Journal of Electrical Power & Energy Systems. 2023, 153: 109292. doi: 10.1016/j.ijepes.2023.109292
74. Xiong K, Hu W, Cao D, et al. Coordinated energy management strategy for multi-energy hub with thermo-electrochemical effect based power-to-ammonia: A multi-agent deep reinforcement learning enabled approach. Renewable Energy. 2023, 214: 216-232. doi: 10.1016/j.renene.2023.05.067
75. Guo G, Zhang M, Gong Y, et al. Safe multi-agent deep reinforcement learning for real-time decentralized control of inverter based renewable energy resources considering communication delay. Applied Energy. 2023, 349: 121648. doi: 10.1016/j.apenergy.2023.121648
76. Li Y, Bu F, Li Y, et al. Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: A deep reinforcement learning approach. Applied Energy. 2023, 333: 120540. doi: 10.1016/j.apenergy.2022.120540
77. Zhong S, Wang X, Zhao J, et al. Deep reinforcement learning framework for dynamic pricing demand response of regenerative electric heating. Applied Energy. 2021, 288: 116623. doi: 10.1016/j.apenergy.2021.116623
78. Li J, Yu T, Zhang X. Coordinated load frequency control of multi-area integrated energy system using multi-agent deep reinforcement learning. Applied Energy. 2022, 306: 117900. doi: 10.1016/j.apenergy.2021.117900
79. Ye Y, Qiu D, Wu X, et al. Model-Free Real-Time Autonomous Control for a Residential Multi-Energy System Using Deep Reinforcement Learning. IEEE Transactions on Smart Grid. 2020, 11(4): 3068-3082. doi: 10.1109/tsg.2020.2976771
80. Zhou S, Hu Z, Gu W, et al. Combined heat and power system intelligent economic dispatch: A deep reinforcement learning approach. International Journal of Electrical Power & Energy Systems. 2020, 120: 106016. doi: 10.1016/j.ijepes.2020.106016
81. Dong Y, Zhang H, Wang C, et al. Soft actor-critic DRL algorithm for interval optimal dispatch of integrated energy systems with uncertainty in demand response and renewable energy. Engineering Applications of Artificial Intelligence. 2024, 127: 107230. doi: 10.1016/j.engappai.2023.107230
82. Yun L, Wang D, Li L. Explainable multi-agent deep reinforcement learning for real-time demand response towards sustainable manufacturing. Applied Energy. 2023, 347: 121324. doi: 10.1016/j.apenergy.2023.121324
83. Xie J, Ajagekar A, You F. Multi-Agent attention-based deep reinforcement learning for demand response in grid-responsive buildings. Applied Energy. 2023, 342: 121162. doi: 10.1016/j.apenergy.2023.121162
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Jiehui Zheng, Yingying Su, Wenhao Wang, Zhigang Li, Qinghua Wu
This work is licensed under a Creative Commons Attribution 4.0 International License.
