Soft-HGRNs: soft hierarchical graph recurrent networks for multi-agent partially observable environments

Yixiang REN; Zhenhui YE; Yining CHEN; Xiaohong JIANG; Guanghua SONG

doi:10.1631/FITEE.2200073

Your Location：

Home >

Browse articles >

Soft-HGRNs: soft hierarchical graph recurrent networks for multi-agent partially observable environments

Regular Papers | Updated：2023-01-28

- Soft-HGRNs: soft hierarchical graph recurrent networks for multi-agent partially observable environments
- Soft-HGRNs：用于多智能体部分可观察场景的随机性层次图递归网络
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 1, Pages: 117-130(2023)
- Affiliations：
  
  1.School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China
  2.College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
- Author bio：
  
  †E-mail: yixiangren@zju.edu.cn;
  zhenhuiye@zju.edu.cn;
  ch19930611@zju.edu.cn;
  ‡Corresponding authors
- Funds：
- DOI：10.1631/FITEE.2200073
  CLC： TP181
- Received：25 February 2022，
  
  Accepted：11 August 2022，
  
  Published：0 January 2023
- Accepted：
Scan QR Code
Yixiang REN, Zhenhui YE, Yining CHEN, et al. Soft-HGRNs: soft hierarchical graph recurrent networks for multi-agent partially observable environments[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(1): 117-130.
DOI：

Yixiang REN, Zhenhui YE, Yining CHEN, et al. Soft-HGRNs: soft hierarchical graph recurrent networks for multi-agent partially observable environments[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(1): 117-130. DOI： 10.1631/FITEE.2200073.

摘要

近年来，多智能体深度强化学习（multi-agent deep reinforcement learning，MADRL）的研究进展使其在现实世界的任务中更加实用，但其相对较差的可扩展性和部分可观测的限制为MADRL模型的性能和部署带来了更多的挑战。人类社会可以被视为一个大规模的部分可观测环境，其中每个人都具备与他人交流并记忆经验的功能。基于人类社会的启发，我们提出一种新的网络结构，称为层次图递归网络（hierarchical graph recurrent network，HGRN），用于部分可观测环境下的多智能体合作任务。具体来说，我们将多智能体系统构建为一个图，利用新颖的图卷积结构来实现异构相邻智能体之间的通信，并采用一个递归单元来使智能体具备记忆历史信息的能力。为了鼓励智能体探索并提高模型的鲁棒性，我们进而设计一种最大熵学习方法，令智能体可以学习可配置目标行动熵的随机策略。基于上述技术，我们提出一种名为Soft-HGRN的基于值的MADRL算法，及其名为SAC-HGRN的actor-critic变体。在三个同构场景和一个异构环境中进行实验; 实验结果不仅表明我们的方法相比四个MADRL基线取得了明显的改进，而且证明了所提模型的可解释性、可扩展性和可转移性。

Abstract

The recent progress in multi-agent deep reinforcement learning (MADRL) makes it more practical in real-world tasks

but its relatively poor scalability and the partially observable constraint raise more challenges for its performance and deployment. Based on our intuitive observation that human society could be regarded as a large-scale partially observable environment

where everyone has the functions of communicating with neighbors and remembering his/her own experience

we propose a novel network structure called the hierarchical graph recurrent network (HGRN) for multi-agent cooperation under partial observability. Specifically

we construct the multi-agent system as a graph

use a novel graph convolution structure to achieve communication between heterogeneous neighboring agents

and adopt a recurrent unit to enable agents to record historical information. To encourage exploration and improve robustness

we design a maximum-entropy learning method that can learn stochastic policies of a configurable target action entropy. Based on the above technologies

we propose a value-based MADRL algorithm called Soft-HGRN and its actor-critic variant called SAC-HGRN. Experimental results based on three homogeneous tasks and one heterogeneous environment not only show that our approach achieves clear improvements compared with four MADRL baselines

but also demonstrate the interpretability

scalability

and transferability of the proposed model.

关键词

Keywords

references

Adler JL , Satapathy G , Manikonda V , et al. , 2005 . A multi-agent approach to cooperative traffic management and route guidance . Trans Res Part B , 39 ( 4 ): 297 - 318 . doi: 10.1016/j.trb.2004.03.005 http://doi.org/10.1016/j.trb.2004.03.005

Cho K , van Merriënboer B , Gulcehre C , et al. , 2014 . Learning phrase representations using RNN encoder-decoder for statistical machine translation . Proc Conf on Empirical Methods in Natural Language Processing , p. 1724 - 1734 . doi: 10.3115/v1/D14-1179 http://doi.org/10.3115/v1/D14-1179

Chu TS , Wang J , Codecà L , et al. , 2020 . Multi-agent deep reinforcement learning for large-scale traffic signal control . IEEE Trans Intell Transp Syst , 21 ( 3 ): 1086 - 1095 . doi: 10.1109/TITS.2019.2901791 http://doi.org/10.1109/TITS.2019.2901791

Claus C , Boutilier C , 1998 . The dynamics of reinforcement learning in cooperative multiagent systems . Proc 15 th National Conf on Artificial Intelligence and 10 th Innovative Applications of Artificial Intelligence Conf , p. 746 - 752 .

Foerster JN , Assael YM , de Freitas N , et al. , 2016 . Learning to communicate with deep multi-agent reinforcement learning . Proc 30 th Int Conf on Neural Information Processing Systems , p. 2145 - 2153 .

Haarnoja T , Tang HR , Abbeel P , et al. , 2017 . Reinforcement learning with deep energy-based policies . Proc 34 th Int Conf on Machine Learning , p. 1352 - 1361 .

Haarnoja T , Zhou A , Abbeel P , et al. , 2018 . Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor . Proc 35 th Int Conf on Machine Learning , p. 1861 - 1870 .

Hausknecht M , Stone P , 2015 . Deep recurrent Q-learning for partially observable MDPs . Proc AAAI Fall Symp Series , p. 29 - 37 .

He KM , Zhang XY , Ren SQ , et al. , 2016 . Deep residual learning for image recognition . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 770 - 778 . doi: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90

Iqbal S , Sha F , 2019 . Actor-attention-critic for multi-agent reinforcement learning . Proc 36 th Int Conf on Machine Learning , p. 2961 - 2970 .

Jiang JC , Dun C , Huang TJ , et al. , 2020 . Graph convolutional reinforcement learning . Proc 8 th Int Conf on Learning Representations .

Kingma DP , Ba J , 2015 . Adam: a method for stochastic optimization . Proc 3 rd Int Conf on Learning Representations .

Lillicrap TP , Hunt JJ , Pritzel A , et al. , 2015 . Continuous control with deep reinforcement learning . Proc 4 th Int Conf on Learning Representations .

Lin LJ , 1992 . Self-improving reactive agents based on reinforcement learning, planning and teaching . Mach Learn , 8 ( 3-4 ): 293 - 321 . doi: 10.1007/BF00992699 http://doi.org/10.1007/BF00992699

Lowe R , Wu Y , Tamar A , et al. , 2017 . Multi-agent actor-critic for mixed cooperative-competitive environments . Proc 31 st Int Conf on Neural Information Processing Systems , p. 6382 - 6393 .

Mnih V , Kavukcuoglu K , Silver D , et al. , 2015 . Human-level control through deep reinforcement learning . Nature , 518 ( 7540 ): 529 - 533 . doi: 10.1038/nature14236 http://doi.org/10.1038/nature14236

Mnih V , Badia AP , Mirza M , et al. , 2016 . Asynchronous methods for deep reinforcement learning . Proc 33 rd Int Conf on Machine Learning , p. 1928 - 1937 .

Rashid T , Samvelyan M , Schroeder C , et al. , 2018 . QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning . Proc 35 th Int Conf on Machine Learning , p. 4295 - 4304 .

Rui P , 2010 . Multi-UAV formation maneuvering control based on Q-learning fuzzy controller . Proc 2 nd Int Conf on Advanced Computer Control , p. 252 - 257 . doi: 10.1109/ICACC.2010.5486961 http://doi.org/10.1109/ICACC.2010.5486961

Ryu H , Shin H , Park J , 2020 . Multi-agent actor-critic with hierarchical graph attention network . Proc AAAI Conf Artif Intell , p. 7236 - 7243 . doi: 10.1609/aaai.v34i05.6214 http://doi.org/10.1609/aaai.v34i05.6214

Sukhbaatar S , Szlam A , Fergus R , 2016 . Learning multi- agent communication with backpropagation . Proc 30 th Int Conf on Neural Information Processing Systems , p. 2252 - 2260 .

Vaswani A , Shazeer N , Parmar N , et al. , 2017 . Attention is all you need . Proc 31 st Int Conf on Neural Information Processing Systems , p. 6000 - 6010 . doi: 10.5555/3295222.3295349 http://doi.org/10.5555/3295222.3295349

Veličković P , Cucurull G , Casanova A , et al. , 2018 . Graph attention networks . Proc 6 th Int Conf on Learning Representations .

Wang RE , Everett M , How JP , 2020 . R-MADDPG for partially observable environments and limited communication . doi: 10.48550/arXiv.2002.06684 http://doi.org/10.48550/arXiv.2002.06684

Watkins CJCH , Dayan P , 1992 . Q -learning . Mach Learn , 8 ( 3-4 ): 279 - 292 . doi: 10.1007/BF00992698 http://doi.org/10.1007/BF00992698

Ye DY , Zhang MJ , Yang Y , 2015 . A multi-agent framework for packet routing in wireless sensor networks . Sensors , 15 ( 5 ): 10026 - 10047 . doi: 10.3390/s150510026 http://doi.org/10.3390/s150510026

Ye ZH , Chen YN , Jiang XH , et al. , 2022a . Improving sample efficiency in multi-agent actor-critic methods . Appl Intell , 52 ( 4 ): 3691 - 3704 . doi: 10.1007/s10489-021-02554-5 http://doi.org/10.1007/s10489-021-02554-5

Ye ZH , Wang K , Chen YN , et al. , 2022b . Multi-UAV navigation for partially observable communication coverage by graph reinforcement learning . IEEE Trans Mobile Comput , early access . doi: 10.1109/TMC.2022.3146881 http://doi.org/10.1109/TMC.2022.3146881

Zhang KQ , Yang ZR , Liu H , et al. , 2018 . Fully decentralized multi-agent reinforcement learning with networked agents . Proc 35 th Int Conf on Machine Learning , p. 5872 - 5881 .

Zhang Y , Mou ZY , Gao FF , et al. , 2020 . UAV-enabled secure communications by multi-agent deep reinforcement learning . IEEE Trans Veh Technol , 69 ( 10 ): 11599 - 11611 . doi: 10.1109/TVT.2020.3014788 http://doi.org/10.1109/TVT.2020.3014788

Zheng LM , Yang JC , Cai H , et al. , 2018 . MAgent: a many-agent reinforcement learning platform for artificial collective intelligence . Proc 32 nd AAAI Conf on Artificial Intelligence , p. 8222 - 8223 .

Views

196

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Federated deep reinforcement learning based computation offloading in a low Earth orbit satellite edge computing system

Significance extraction based on data augmentation for reinforcement learning

Incentive-based task offloading for digital twins in 6G native artificial intelligence networks: a learning approach

Deep reinforcement learning for near-field wideband beamforming in STAR-RIS networks

Deep reinforcement learning: a survey

Related Author

Qing GUO

Xinyu WANG

Jian WU

Min JIA

Yang YANG

Dequan LI

Yuxi HAN

Qinqin TANG

Related Institution

School of Electronics and Information Engineering, Harbin Institute of Technology

Faculty of Artificial Intelligence, Anhui University of Science and Technology

China Mobile Research Institute

ZGC Institute of Ubiquitous-X Innovation and Applications

China Mobile Communications Group Corporation

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰