Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents

Jian ZHAO; Youpeng ZHAO; Weixun WANG; Mingyu YANG; Xunhan HU; Wengang ZHOU; Jianye HAO; Houqiang LI

doi:10.1631/FITEE.2100594

Your Location：

Home >

Browse articles >

Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents

Special Feature on Institutionalized and Systematized Gaming (InSys Gaming) for Multi-agent Systems | Updated：2022-09-08

- Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents
- 针对意外崩溃智能体的教练辅助多智能体强化学习框架
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 7, Pages: 1032-1042(2022)
- Affiliations：
  
  1.School of Information Science and Technology, University of Science and Technology of China, Hefei230026, China
  2.College of Intelligence and Computing, Tianjin University, Tianjin300072, China
- Author bio：
  
  E-mail: zj140@mail.ustc.edu.cn;
  ‡Corresponding authors
- Funds：
- DOI：10.1631/FITEE.2100594
  CLC： TP18
- Received：31 December 2021，
  
  Accepted：08 April 2022，
  
  Published：23 July 2022
- Accepted：
Scan QR Code
Jian ZHAO, Youpeng ZHAO, Weixun WANG, et al. Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(7): 1032-1042.
DOI：

Jian ZHAO, Youpeng ZHAO, Weixun WANG, et al. Coach-assisted multi-agent reinforcement learning framework for unexpected crashed agents[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(7): 1032-1042. DOI： 10.1631/FITEE.2100594.

摘要

多智能体强化学习在实际场景中很难应用，一部分原因在于模拟环境和现实环境之间存在差距。造成这种差距的一个原因是，模拟系统总是假设智能体可以一直正常工作，而实际上，由于不可避免的硬件或软件故障，一个或多个智能体可能会在合作过程中意外 “崩溃”。这样的崩溃会破坏智能体之间的合作，导致系统性能下降。本文中，我们给出了意外崩溃情况下合作多智能体强化学习系统的正式定义。为增强系统应对崩溃时的鲁棒性，提出教练辅助多智能体强化学习框架，其在训练过程中引入一个虚拟教练智能体，以调整系统的崩溃概率。为教练智能体设计了3种教练策略和重采样策略。据我们所知，这是研究多智能体系统中意外崩溃情况的首项工作。在网格环境和星际争霸微管理任务上的大量实验表明，相比固定崩溃概率和课程学习的教练策略，自适应策略更加有效。消融实验进一步展现了重采样策略的有效性。

Abstract

Multi-agent reinforcement learning is difficult to apply in practice

partially because of the gap between simulated and real-world scenarios. One reason for the gap is that simulated systems always assume that agents can work normally all the time

while in practice

one or more agents may unexpectedly "crash" during the coordination process due to inevitable hardware or software failures. Such crashes destroy the cooperation among agents and lead to performance degradation. In this work

we present a formal conceptualization of a cooperative multi-agent reinforcement learning system with unexpected crashes. To enhance the robustness of the system to crashes

we propose a coach-assisted multi-agent reinforcement learning framework that introduces a virtual coach agent to adjust the crash rate during training. We have designed three coaching strategies (fixed crash rate

curriculum learning

and adaptive crash rate) and a re-sampling strategy for our coach agent. To our knowledge

this work is the first to study unexpected crashes in a multi-agent system. Extensive experiments on grid-world and StarCraft II micromanagement tasks demonstrate the efficacy of the adaptive strategy compared with the fixed crash rate strategy and curriculum learning strategy. The ablation study further illustrates the effectiveness of our re-sampling strategy.

关键词

Keywords

references

Arndt K , Hazara M , Ghadirzadeh A , et al. , 2020 . Meta reinforcement learning for sim-to-real domain adaptation . Proc IEEE Int Conf on Robotics and Automation , p. 2725 - 2731 . doi: 10.1109/ICRA40945.2020.9196540 http://doi.org/10.1109/ICRA40945.2020.9196540

Busoniu L , Babuska R , de Schutter B , 2008 . A comprehensive survey of multiagent reinforcement learning . IEEE Trans Syst Man Cybern Part C Appl Rev , 38 ( 2 ): 156 - 172 . doi: 10.1109/TSMCC.2007.913919 http://doi.org/10.1109/TSMCC.2007.913919

Dosovitskiy A , Ros G , Codevilla F , et al. , 2017 . Carla: an open urban driving simulator . Proc 1 st Conf on Robot Learning , p. 1 - 16 .

Foerster J , Nardelli N , Farquhar G , et al. , 2017 . Stabilising experience replay for deep multi-agent reinforcement learning . Proc 34 th Int Conf on Machine Learning , p. 1146 - 1155 . doi: 10.5555/3305381.3305500 http://doi.org/10.5555/3305381.3305500

Furrer F , Burri M , Achtelik M , et al. , 2016 . RotorS—a modular Gazebo MAV simulator framework . In: Koubaa A (Ed.), Robot Operating System (ROS): the Complete Reference . Volume 1 , Springer , Cham , p. 595 - 625 . doi: 10.1007/978-3-319-26054-9_23 http://doi.org/10.1007/978-3-319-26054-9_23

Guestrin C , Koller D , Parr R , 2001 . Multiagent planning with factored MDPs . Proc 14 th Int Conf on Neural Information Processing Systems: Natural and Synthetic , p. 1523 - 1530 . doi: 10.5555/2980539.2980737 http://doi.org/10.5555/2980539.2980737

Higgins I , Pal A , Rusu A , et al. , 2017 . DARLA: improving zero-shot transfer in reinforcement learning . Proc 34 th Int Conf on Machine Learning , p. 1480 - 1490 .

Kim D , Moon S , Hostallero D , et al. , 2019 . Learning to schedule communication in multi-agent reinforcement learning . Proc 7 th Int Conf on Learning Representations , p. 1 - 17 .

Kok JR , Vlassis N , 2006 . Collaborative multiagent reinforcement learning by payoff propagation . J Mach Learn Res , 7 : 1789 - 1828 . doi: 10.5555/1248547.1248612 http://doi.org/10.5555/1248547.1248612

Kraemer L , Banerjee B , 2016 . Multi-agent reinforcement learning as a rehearsal for decentralized planning . Neurocomputing , 190 : 82 - 94 . doi: 10.1016/j.neucom.2016.01.031 http://doi.org/10.1016/j.neucom.2016.01.031

Lowe R , Wu Y , Tamar A , et al. , 2017 . Multi-agent actor-critic for mixed cooperative-competitive environments . Proc 31 st Int Conf on Neural Information Processing Systems , p. 6382 - 6393 . doi: 10.5555/3295222.3295385 http://doi.org/10.5555/3295222.3295385

McCord C , Queralta JP , Gia TN , et al. , 2019 . Distributed progressive formation control for multi-agent systems: 2D and 3D deployment of UAVs in ROS/Gazebo with rotors . Proc European Conf on Mobile Robots , p. 1 - 6 . doi: 10.1109/ECMR.2019.8870934 http://doi.org/10.1109/ECMR.2019.8870934

Mnih V , Kavukcuoglu K , Silver D , et al. , 2015 . Human-level control through deep reinforcement learning . Nature , 518 ( 7540 ): 529 - 533 . doi: 10.1038/nature14236 http://doi.org/10.1038/nature14236

Narvekar S , Peng B , Leonetti M , et al. , 2020 . Curriculum learning for reinforcement learning domains: a framework and survey . J Mach Learn Res , 21 ( 181 ): 1 - 50 .

Oliehoek FA , Spaan MTJ , Vlassis N , 2008 . Optimal and approximate Q-value functions for decentralized POMDPs . J Artif Intell Res , 32 : 289 - 353 .

Omidshafiei S , Pazis J , Amato C , et al. , 2017 . Deep decentralized multi-task multi-agent reinforcement learning under partial observability . Proc 34 th Int Conf on Machine Learning , p. 2681 - 2690 .

Peng P , Wen Y , Yang YD , et al. , 2017 . Multiagent bidirectionally-coordinated nets: emergence of human-level coordination in learning to play starcraft combat games . Proc 34 th Int Conf on Machine Learning , p. 2681 - 2690 .

Rashid T , Samvelyan M , de Witt SC , et al. , 2018 . QMIX: monotonic value function factorisation for deep multi-agent reinforcement learning . Proc 35 th Int Conf on Machine Learning , p. 4292 - 4301 .

Samvelyan M , Rashid T , de Witt CS , et al. , 2019 . The StarCraft Multi-agent Challenge . Proc 18 th Int Conf on Autonomous Agents and Multiagent Systems , p. 2186 - 2188 . doi: 10.5555/3306127.3332052 http://doi.org/10.5555/3306127.3332052

Shah S , Dey D , Lovett C , et al. , 2018 . AirSim: high-fidelity visual and physical simulation for autonomous vehicles . 11 th Int Conf on Field and Service Robotics , p. 621 - 635 . doi: 10.1007/978-3-319-67361-5_40 http://doi.org/10.1007/978-3-319-67361-5_40

Son K , Kim D , Kang WJ , et al. , 2019 . QTRAN: learning to factorize with transformation for cooperative multi-agent reinforcement learning . Proc 36 th Int Conf on Machine Learning , p. 5887 - 5896 .

Sukhbaatar S , Szlam A , Fergus R , 2016 . Learning multi-agent communication with backpropagation . Proc 30 th Int Conf on Neural Information Processing Systems , p. 2252 - 2260 . doi: 10.5555/3157096.3157348 http://doi.org/10.5555/3157096.3157348

Sunehag P , Lever G , Gruslys A , et al. , 2018 . Value-decomposition networks for cooperative multi-agent learning based on team reward . Proc 17 th Int Conf on Autonomous Agents and Multiagent Systems , p. 2085 - 2087 . doi: 10.5555/3237383.3238080 http://doi.org/10.5555/3237383.3238080

Tan M , 1993 . Multi-agent Reinforcement Learning: Independent vs. Cooperative Agents . Morgan Kaufmann , p. 330 - 337 . doi: 10.1016/B978-1-55860-307-3.50049-6 http://doi.org/10.1016/B978-1-55860-307-3.50049-6

Tobin J , Fong R , Ray A , et al. , 2017 . Domain randomization for transferring deep neural networks from simulation to the real world . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , p. 23 - 30 . doi: 10.1109/IROS.2017.8202133 http://doi.org/10.1109/IROS.2017.8202133

Todorov E , Erez T , Tassa Y , 2012 . MuJoCo: a physics engine for model-based control . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , p. 5026 - 5033 . doi: 10.1109/IROS.2012.6386109 http://doi.org/10.1109/IROS.2012.6386109

Traoré R , Caselles-Dupré H , Lesort T , et al. , 2019 . Continual reinforcement learning deployed in real-life using policy distillation and Sim2Real transfer . https://arxiv.org/abs/1906.04452 https://arxiv.org/abs/1906.04452

Tuyls K , Weiss G , 2012 . Multiagent learning: basics, challenges, and prospects . AI Mag , 33 ( 3 ): 41 . doi: 10.1609/aimag.v33i3.2426 http://doi.org/10.1609/aimag.v33i3.2426

Wang JH , Ren ZZ , Liu T , et al. , 2020 . QPLEX: duplex dueling multi-agent Q-learning . Proc 9 th Int Conf on Learning Representations , p. 1 - 16 .

Wang YP , Zheng KX , Tian DX , et al. , 2020 . Cooperative channel assignment for VANETs based on multiagent reinforcement learning . Front Inform Technol Electron Eng , 21 ( 7 ): 1047 - 1058 . doi: 10.1631/FITEE.1900308 http://doi.org/10.1631/FITEE.1900308

Wang YP , Zheng KX , Tian DX , et al. , 2021 . Pre-training with asynchronous supervised learning for reinforcement learning based autonomous driving . Front Inform Technol Electron Eng , 22 ( 5 ): 673 - 686 . doi: 10.1631/FITEE.1900637 http://doi.org/10.1631/FITEE.1900637

Yang YD , Hao JY , Liao B , et al. , 2020 . QATTEN: a general framework for cooperative multiagent reinforcement learning . https://arxiv.org/abs/2002.03939 https://arxiv.org/abs/2002.03939

Zhang KQ , Yang ZR , Basar T , 2021 . Decentralized multi-agent reinforcement learning with networked agents: recent advances . Front Inform Technol Electron Eng , 22 ( 6 ): 802 - 814 . doi: 10.1631/FITEE.1900661 http://doi.org/10.1631/FITEE.1900661

Zhao WS , Queralta JP , Westerlund T , 2020 . Sim-to-real transfer in deep reinforcement learning for robotics: a survey . Proc IEEE Symp Series on Computational Intelligence , p. 737 - 744 . doi: 10.1109/SSCI47803.2020.9308468 http://doi.org/10.1109/SSCI47803.2020.9308468

Views

101

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Multi-agent differential game based cooperative synchronization control using a data-driven method

Jiu fusion artificial intelligence (JFA): a two-stage reinforcement learning model with hierarchical neural networks and human knowledge for Tibetan Jiu chess

Efficient learning of robust multigait quadruped locomotion for minimizing the cost of transport

Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Related Author

Yu SHI

Yongzhao HUA

Jianglong YU

Xiwang DONG

Zhang REN

Xiali LI

Xiaoyu FAN

Junzhi YU

Related Institution

School of Automation Science and Electrical Engineering, Beihang University

Institute of Artificial Intelligence, Beihang University

School of Information Science and Technology, Xizang University

State Key Laboratory of General Artificial Intelligence, Peking University

State Key Laboratory for Turbulence and Complex Systems, Peking University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰