基于解耦价值和策略强化学习的家庭能源管理方法

熊珞琳; 唐漾; 刘臣胜; 毛帅; 孟科; 董朝阳; 钱锋

doi:10.1631/FITEE.2200667

Your Location：

Home >

Browse articles >

基于解耦价值和策略强化学习的家庭能源管理方法

专题 | Updated：2023-09-21

- 基于解耦价值和策略强化学习的家庭能源管理方法
  Enhanced Publication
- A home energy management approach using decoupling value and policy in reinforcement learning
- 信息与电子工程前沿（英文） 2023年24卷第9期页码：1261-1272
- Affiliations：
  
  1.Key Laboratory of Smart Manufacturing in Energy Chemical Process, Ministry of Education, East China University of Science and Technology, Shanghai 200237, China
  2.Department of Electrical Engineering, Nantong University, Nantong 226019, China
  3.School of Electrical Engineering and Telecommunications, University of New South Wales, SydneyNSW 2052, Australia
  4.School of Electrical and Electronics Engineering, Nanyang Technological University, Singapore 639798, Singapore
- Author bio：
  
  ‡Corresponding authors
- Funds：
  
  National Natural Science Foundation of China(62293502;62293500;62293504;62073138;62173147);Fundamental Research Funds for the Central Universities, China(222202317006);Nanyang Technological University Startup Grant and MOE Tier 1(RG59/22)
- DOI：10.1631/FITEE.2200667
  中图分类号： TP181
- 纸质出版日期：2023-09-0 ，
  
  网络出版日期：2023-08-10，
  
  收稿日期：2022-12-27，
  
  录用日期：2023-04-09
- Accepted：
Scan QR Code
熊珞琳, 唐漾, 刘臣胜, 等. 基于解耦价值和策略强化学习的家庭能源管理方法[J]. 信息与电子工程前沿（英文）, 2023,24(9):1261-1272.

LUOLIN XIONG, YANG TANG, CHENSHENG LIU, et al. A home energy management approach using decoupling value and policy in reinforcement learning. [J]. Frontiers of information technology & electronic engineering, 2023, 24(9): 1261-1272.
熊珞琳, 唐漾, 刘臣胜, 等. 基于解耦价值和策略强化学习的家庭能源管理方法[J]. 信息与电子工程前沿（英文）, 2023,24(9):1261-1272. DOI： 10.1631/FITEE.2200667.

LUOLIN XIONG, YANG TANG, CHENSHENG LIU, et al. A home energy management approach using decoupling value and policy in reinforcement learning. [J]. Frontiers of information technology & electronic engineering, 2023, 24(9): 1261-1272. DOI： 10.1631/FITEE.2200667.

摘要

由于电动汽车的普及性和家用电器的灵活性，在动态电价下对家庭能源系统进行能源调度优化电力成本和保障居民舒适度是可行的。本文提出一种基于数据驱动的深度强化学习家庭能源管理方法。首先，为揭示影响电动汽车充电行为的多种不确定因素，引入一种结合驾驶员经验、突发事件和交通状况的改进数学模型描述电动汽车在家庭能源系统中的动态能量需求。其次，提出一种解耦优势演员-评论家（DA2C）算法，通过缓解策略和价值共享网络导致的过拟合问题提升能源优化性能。此外，策略函数和价值函数的解耦网络确保了所提方法在不可见场景中的泛化性。最后，将所提方法与现有方法进行综合实验比较。结果表明，该方法能在不同场景下优化用电成本并兼顾居住舒适度。

Abstract

Considering the popularity of electric vehicles and the flexibility of household appliances

it is feasible to dispatch energy in home energy systems under dynamic electricity prices to optimize electricity cost and comfort residents. In this paper

a novel home energy management (HEM) approach is proposed based on a data-driven deep reinforcement learning method. First

to reveal the multiple uncertain factors affecting the charging behavior of electric vehicles (EVs)

an improved mathematical model integrating driver’s experience

unexpected events

and traffic conditions is introduced to describe the dynamic energy demand of EVs in home energy systems. Second

a decoupled advantage actor-critic (DA2C) algorithm is presented to enhance the energy optimization performance by alleviating the overfitting problem caused by the shared policy and value networks. Furthermore

separate networks for the policy and value functions ensure the generalization of the proposed method in unseen scenarios. Finally

comprehensive experiments are carried out to compare the proposed approach with existing methods

and the results show that the proposed method can optimize electricity cost and consider the residential comfort level in different scenarios.

关键词

家庭能源系统电动汽车强化学习泛化性

Keywords

Home energy systemElectric vehicleReinforcement learningGeneralization

references

Agnew S, Dargusch P, 2015. Effect of residential solar and storage on centralized electricity supply systems. Nat Climate Change, 5（4）:315-318. doi: 10.1038/nclimate2523http://doi.org/10.1038/nclimate2523

Anvari-Moghaddam A, Monsef H, Rahimi-Kian A, 2015. Optimal smart home energy management considering energy saving and a comfortable lifestyle. IEEE Trans Smart Grid, 6（1）:324-332. doi: 10.1109/TSG.2014.2349352http://doi.org/10.1109/TSG.2014.2349352

Ausgrid , 2014. Solar Home Electricity Data. http://www.ipart.nsw.gov.auhttp://www.ipart.nsw.gov.au ［Accessed on Nov. 30, 2022］.

Baek K, Lee E, Kim J, 2021. Resident behavior detection model for environment responsive demand response. IEEE Trans Smart Grid, 12（5）:3980-3989. doi: 10.1109/TSG.2021.3074955http://doi.org/10.1109/TSG.2021.3074955

Cobbe K, Hilton J, Klimov O, et al., 2021. Phasic policy gradient. Proc 38th Int Conf on Machine Learning, p.2020-2027.

Gao HJ, Li ZK, Yu XH, et al., 2022. Hierarchical multiobjective heuristic for PCB assembly optimization in a beam-head surface mounter. IEEE Trans Cybern, 52（7）:6911-6924. doi: 10.1109/TCYB.2020.3040788http://doi.org/10.1109/TCYB.2020.3040788

Hu KY, Li WJ, Wang LD, et al., 2018. Energy management for multi-microgrid system based on model predictive control. Front Inform Technol Electron Eng, 19（11）:1340-1351. doi: 10.1631/FITEE.1601826http://doi.org/10.1631/FITEE.1601826

Huang G, Wu F, Guo CX, 2022. Smart grid dispatch powered by deep learning: a survey. Front Inform Technol Electron Eng, 23（5）:763-776. doi: 10.1631/FITEE.2000719http://doi.org/10.1631/FITEE.2000719

Kong WC, Luo FJ, Jia YW, et al., 2021. Benefits of home energy storage utilization: an Australian case study of demand charge practices in residential sector. IEEE Trans Smart Grid, 12（4）:3086-3096. doi: 10.1109/TSG.2021.3054126http://doi.org/10.1109/TSG.2021.3054126

Kumari A, Tanwar S, 2022. A reinforcement-learning-based secure demand response scheme for smart grid system. IEEE Internet Things J, 9（3）:2180-2191. doi: 10.1109/JIOT.2021.3090305http://doi.org/10.1109/JIOT.2021.3090305

Li HP, Wan ZQ, He HB, 2020a. A deep reinforcement learning based approach for home energy management system. Proc IEEE Power & Energy Society Innovative Smart Grid Technologies Conf, p.1-5. doi: 10.1109/ISGT45199.2020.9087647http://doi.org/10.1109/ISGT45199.2020.9087647

Li HP, Wan ZQ, He HB, 2020b. Real-time residential demand response. IEEE Trans Smart Grid, 11（5）:4144-4154. doi: 10.1109/TSG.2020.2978061http://doi.org/10.1109/TSG.2020.2978061

Li JH, 2018. Cyber security meets artificial intelligence: a survey. Front Inform Technol Electron Eng, 19（12）:1462-1474. doi: 10.1631/FITEE.1800573http://doi.org/10.1631/FITEE.1800573

Liu SG, Zheng SZ, Zhang WB, et al., 2022. A power resource dispatching framework with a privacy protection function in the power Internet of Things. Front Inform Technol Electron Eng, 23（9）:1354-1368. doi: 10.1631/FITEE.2100518http://doi.org/10.1631/FITEE.2100518

Liu YB, Liu JY, Taylor G, et al., 2016. Situational awareness architecture for smart grids developed in accordance with dispatcher’s thought process: a review. Front Inform Technol Electron Eng, 17（11）:1107-1121. doi: 10.1631/FITEE.1601516http://doi.org/10.1631/FITEE.1601516

Liu ZT, Lin WY, Yu XH, et al., 2022. Approximation-free robust synchronization control for dual-linear-motors-driven systems with uncertainties and disturbances. IEEE Trans Ind Electron, 69（10）:10500-10509. doi: 10.1109/TIE.2021.3137619http://doi.org/10.1109/TIE.2021.3137619

Liu ZT, Gao HJ, Yu XH, et al., 2023. B-spline wavelet neural network-based adaptive control for linear motor-driven systems via a novel gradient descent algorithm. IEEE Trans Ind Electron, early access. doi: 10.1109/TIE.2023.3260318http://doi.org/10.1109/TIE.2023.3260318

Lu RZ, Jiang ZY, Wu HM, et al., 2023. Reward shaping-based actor-critic deep reinforcement learning for residential energy management. IEEE Trans Ind Inform, 19（3）:2662-2673. doi: 10.1109/TII.2022.3183802http://doi.org/10.1109/TII.2022.3183802

Luo FJ, Kong WC, Ranzi G, et al., 2020. Optimal home energy management system with demand charge tariff and appliance operational dependencies. IEEE Trans Smart Grid, 11（1）:4-14. doi: 10.1109/TSG.2019.2915679http://doi.org/10.1109/TSG.2019.2915679

Mao S, Wang B, Tang Y, et al., 2019. Opportunities and challenges of artificial intelligence for green manufacturing in the process industry. Engineering, 5（6）:995-1002. doi: 10.1016/j.eng.2019.08.013http://doi.org/10.1016/j.eng.2019.08.013

Mao S, Tang Y, Dong ZW, et al., 2021. A privacy preserving distributed optimization algorithm for economic dispatch over time-varying directed networks. IEEE Trans Ind Inform, 17（3）:1689-1701. doi: 10.1109/TII.2020.2996198http://doi.org/10.1109/TII.2020.2996198

Nafi NM, Glasscock C, Hsu W, 2022. Attention-based partial decoupling of policy and value for generalization in reinforcement learning. Proc 21st IEEE Int Conf on Machine Learning and Applications, p.15-22. doi: 10.1109/ICMLA55696.2022.00011http://doi.org/10.1109/ICMLA55696.2022.00011

ta K, Oiki T, Jha D, et al., 2020. Can increasing input dimensionality improve deep reinforcement learning? Proc 37th Int Conf on Machine Learning, p.7424-7433.

Parag Y, Sovacool BK, 2016. Electricity market design for the prosumer era. Nat Energy, 1（4）:16032. doi: 10.1038/nenergy.2016.32http://doi.org/10.1038/nenergy.2016.32

Qian F, 2019. Smart process manufacturing systems: deep integration of artificial intelligence and process manufacturing. Engineering, 5（6）:981. doi: 10.1016/j.eng.2019.10.002http://doi.org/10.1016/j.eng.2019.10.002

Qian F, 2021. Editorial for special issue “artificial intelligence energizes process manufacturing”. Engineering, 7（9）:1193-1194. doi: 10.1016/j.eng.2021.08.003http://doi.org/10.1016/j.eng.2021.08.003

Qin ZM, Liu D, Hua HC, et al., 2021. Privacy preserving load control of residential microgrid via deep reinforcement learning. IEEE Trans Smart Grid, 12（5）:4079-4089. doi: 10.1109/TSG.2021.3088290http://doi.org/10.1109/TSG.2021.3088290

Raileanu R, Fergus R, 2021. Decoupling value and policy for generalization in reinforcement learning. Proc 38th Int Conf on Machine Learning, p.8787-8798.

Rastegar M, Fotuhi-Firuzabad M, Zareipour H, et al., 2017. A probabilistic energy management scheme for renewable-based residential energy hubs. IEEE Trans Smart Grid, 8（5）:2217-2227. doi: 10.1109/TSG.2016.2518920http://doi.org/10.1109/TSG.2016.2518920

Saberi H, Zhang C, Dong ZY, 2021. Data-driven distributionally robust hierarchical coordination for home energy management. IEEE Trans Smart Grid, 12（5）:4090-4101. doi: 10.1109/TSG.2021.3088433http://doi.org/10.1109/TSG.2021.3088433

Schulman J, Wolski F, Dhariwal P, et al., 2017. Proximal policy optimization algorithms. https://arxiv.org/abs/1707.06347https://arxiv.org/abs/1707.06347

Shi PW, Sun WC, Yang XB, et al., 2023. Master-slave synchronous control of dual-drive gantry stage with cogging force compensation. IEEE Trans Syst Man Cybern Syst, 53（1）:216-225. doi: 10.1109/TSMC.2022.3176952http://doi.org/10.1109/TSMC.2022.3176952

Shirsat A, Tang WY, 2021. Data-driven stochastic model predictive control for DC-coupled residential PV-storage systems. IEEE Trans Energy Convers, 36（2）:1435-1448. doi: 10.1109/TEC.2021.3061360http://doi.org/10.1109/TEC.2021.3061360

Shuvo SS, Yilmaz Y, 2022. Home energy recommendation system （HERS）: a deep reinforcement learning method based on residents’ feedback and activity. IEEE Trans Smart Grid, 13（4）:2812-2821. doi: 10.1109/TSG.2022.3158814http://doi.org/10.1109/TSG.2022.3158814

Tang Y, Zhao C, Wang J, et al., 2022. Perception and navigation in autonomous systems in the era of learning: a survey. IEEE Trans Neur Netw Learn Syst, early access. doi: 10.1109/TNNLS.2022.3167688http://doi.org/10.1109/TNNLS.2022.3167688

Wang AJ, Liu WP, Dong T, et al., 2022. DisEHPPC: enabling heterogeneous privacy-preserving consensus-based scheme for economic dispatch in smart grids. IEEE Trans Cybern, 52（6）:5124-5135. doi: 10.1109/TCYB.2020.3027572http://doi.org/10.1109/TCYB.2020.3027572

Wang HN, Liu N, Zhang YY, et al., 2020. Deep reinforcement learning: a survey. Front Inform Technol Electron Eng, 21（12）:1726-1744. doi: 10.1631/FITEE.1900533http://doi.org/10.1631/FITEE.1900533

Wang JR, Hong YT, Wang JL, et al., 2022. Cooperative and competitive multi-agent systems: from optimization to games. IEEE/CAA J Autom Sin, 9（5）:763-783. doi: 10.1109/JAS.2022.105506http://doi.org/10.1109/JAS.2022.105506

Wang YP, Zheng KX, Tian DX, et al., 2021. Pre-training with asynchronous supervised learning for reinforcement learning based autonomous driving. Front Inform Technol Electron Eng, 22（5）:673-686. doi: 10.1631/FITEE.1900637http://doi.org/10.1631/FITEE.1900637

Wen GH, Yu XH, Liu ZW, 2021. Recent progress on the study of distributed economic dispatch in smart grid: an overview. Front Inform Technol Electron Eng, 22（1）:25-39. doi: 10.1631/FITEE.2000205http://doi.org/10.1631/FITEE.2000205

Xia YH, Liu JY, Huang ZW, et al., 2016. Carbon emission impact on the operation of virtual power plant with combined heat and power system. Front Inform Technol Electron Eng, 17（5）:479-488. doi: 10.1631/FITEE.1500467http://doi.org/10.1631/FITEE.1500467

Xiong LL, Mao S, Tang Y, et al., 2021. Reinforcement learning based integrated energy system management: a survey. Acta Autom Sin, 47（10）:2321-2340（in Chinese）. doi: 10.16383/j.aas.c210166http://doi.org/10.16383/j.aas.c210166

Xiong LL, Tang Y, Mao S, et al., 2022. A two-level energy management strategy for multi-microgrid systems with interval prediction and reinforcement learning. IEEE Trans Circ Syst I Regul Pap, 69（4）:1788-1799. doi: 10.1109/TCSI.2022.3141229http://doi.org/10.1109/TCSI.2022.3141229

Xu X, Jia YW, Xu Y, et al., 2020. A multi-agent reinforcement learning-based data-driven method for home energy management. IEEE Trans Smart Grid, 11（4）:3201-3211. doi: 10.1109/TSG.2020.2971427http://doi.org/10.1109/TSG.2020.2971427

Yan LF, Chen X, Zhou JY, et al., 2021. Deep reinforcement learning for continuous electric vehicles charging control with dynamic user behaviors. IEEE Trans Smart Grid, 12（6）:5124-5134. doi: 10.1109/TSG.2021.3098298http://doi.org/10.1109/TSG.2021.3098298

Zenginis I, Vardakas J, Koltsaklis NE, et al., 2022. Smart home’s energy management through a clustering-based reinforcement learning approach. IEEE Internet Things J, 9（17）:16363-16371. doi: 10.1109/JIOT.2022.3152586http://doi.org/10.1109/JIOT.2022.3152586

Zhang HF, Yue D, Dou CX, et al., 2022. Two-layered hierarchical optimization strategy with distributed potential game for interconnected hybrid energy systems. IEEE Trans Cybern, early access. doi: 10.1109/TCYB.2022.3142035http://doi.org/10.1109/TCYB.2022.3142035

Zhang YA, Yang QY, An D, et al., 2022. Multistep multi-agent reinforcement learning for optimal energy schedule strategy of charging stations in smart grid. IEEE Trans Cybern, 53（7）:4292-4305. doi: 10.1109/TCYB.2022.3165074http://doi.org/10.1109/TCYB.2022.3165074

Zhang YI, Ai ZY, Chen JC, et al., 2022. Energy-saving optimization and control of autonomous electric vehicles with considering multiconstraints. IEEE Trans Cybern, 52（10）:10869-10881. doi: 10.1109/TCYB.2021.3069674http://doi.org/10.1109/TCYB.2021.3069674

Zhou SP, 2020. Summary of Time of Use Electricity Price Policy at Home and Abroad（in Chinese）. https://shoudian.bjx.com.cn/html/20200807/1095247.shtmlhttps://shoudian.bjx.com.cn/html/20200807/1095247.shtml ［Accessed on Nov. 30, 2022］.

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Multi-agent reinforcement learning behavioral control for nonlinear second-order systems

An anti-collision algorithm for robotic search-and-rescue tasks in unknown dynamic environments

Embedding expert demonstrations into clustering buffer for effective deep reinforcement learning

A multipath routing algorithm for satellite networksbased on service demand and traffic awareness