Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game

Hongyang LI; Qinglai WEI

doi:10.1631/FITEE.2200010

Your Location：

Home >

Browse articles >

Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game

Special Feature on Institutionalized and Systematized Gaming (InSys Gaming) for Multi-agent Systems | Updated：2022-09-08

- Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game
- 输入饱和下多智能体系统最优一致性控制：一类非零和博弈方法
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 7, Pages: 1010-1019(2022)
- Affiliations：
  
  1.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing100049, China
  2.The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing100190, China
  3.Institute of Systems Engineering, Macau University of Science and Technology, Macau999078, China
- Author bio：
  
  E-mail: lihongyang2019@ia.ac.cn;
  ‡Corresponding authors
- Funds：
- DOI：10.1631/FITEE.2200010
  CLC： TP13
- Received：14 January 2022，
  
  Accepted：07 March 2022，
  
  Published Online：20 May 2022，
  
  Published：23 July 2022
- Accepted：
Scan QR Code
Hongyang LI, Qinglai WEI. Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(7): 1010-1019.
DOI：

Hongyang LI, Qinglai WEI. Optimal synchronization control for multi-agent systems with input saturation: a nonzero-sum game[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(7): 1010-1019. DOI： 10.1631/FITEE.2200010.

摘要

本文针对输入饱和下的多智能体系统，提出一种最优一致性控制方法。引入多智能体博弈理论，将最优一致性控制问题转化为多智能体非零和博弈。之后，通过求解具有非二次输入能量项的耦合Hamilton–Jacobi–Bellman（HJB）方程，实现Nash平衡。提出脱策强化学习方法，在系统模型未知情况下获得Nash平衡解；引入评判神经网络和执行神经网络实现所提方法。理论分析显示迭代控制律收敛到Nash平衡。仿真实验验证了所提方法的有效性。

Abstract

This paper presents a novel optimal synchronization control method for multi-agent systems with input saturation. The multi-agent game theory is introduced to transform the optimal synchronization control problem into a multi-agent nonzero-sum game. Then

the Nash equilibrium can be achieved by solving the coupled Hamilton–Jacobi–Bellman (HJB) equations with nonquadratic input energy terms. A novel off-policy reinforcement learning method is presented to obtain the Nash equilibrium solution without the system models

and the critic neural networks (NNs) and actor NNs are introduced to implement the presented method. Theoretical analysis is provided

which shows that the iterative control laws converge to the Nash equilibrium. Simulation results show the good performance of the presented method.

关键词

Keywords

references

Abu-Khalaf M , Lewis FL , 2005 . Nearly optimal control laws for nonlinear systems with saturating actuators using a neural network HJB approach . Automatica , 41 ( 5 ): 779 - 791 . doi: 10.1016/j.automatica.2004.11.034 http://doi.org/10.1016/j.automatica.2004.11.034

Bertsekas DP , 2007 . Dynamic Programming and Optimal Control . Athena Scientific , Belmont, USA.

Cao MT , Xiao F , Wang L , 2015 . Event-based second-order consensus control for multi-agent systems via synchronous periodic event detection . IEEE Trans Autom Contr , 60 ( 9 ): 2452 - 2457 . doi: 10.1109/TAC.2015.2390553 http://doi.org/10.1109/TAC.2015.2390553

Du HB , He YG , Cheng YY , 2014 . Finite-time synchronization of a class of second-order nonlinear multi-agent systems using output feedback control . IEEE Trans Circ Syst I , 61 ( 6 ): 1778 - 1788 . doi: 10.1109/TCSI.2013.2295012 http://doi.org/10.1109/TCSI.2013.2295012

Garcia E , Cao YC , Casbeer D , 2017 . Periodic event-triggered synchronization of linear multi-agent systems with communication delays . IEEE Trans Autom Contr , 62 ( 1 ): 366 - 371 . doi: 10.1109/TAC.2016.2555484 http://doi.org/10.1109/TAC.2016.2555484

Han YJ , Lu WL , Chen TP , 2013 . Cluster consensus in discrete-time networks of multi-agents with inter-cluster nonidentical inputs . IEEE Trans Neur Netw Learn Syst , 24 ( 4 ): 566 - 578 . doi: 10.1109/TNNLS.2013.2237786 http://doi.org/10.1109/TNNLS.2013.2237786

He WL , Gao XY , Zhong WM , et al. , 2018 . Secure impulsive synchronization control of multi-agent systems under deception attacks . Inform Sci , 459 : 354 - 368 . doi: 10.1016/j.ins.2018.04.020 http://doi.org/10.1016/j.ins.2018.04.020

Jiao Q , Modares H , Xu SY , et al. , 2016 . Multi-agent zero-sum differential graphical games for disturbance rejection in distributed control . Automatica , 69 : 24 - 34 . doi: 10.1016/j.automatica.2016.02.002 http://doi.org/10.1016/j.automatica.2016.02.002

Li JN , Modares H , Chai TY , et al. , 2017 . Off-policy reinforcement learning for synchronization in multiagent graphical games . IEEE Trans Neur Netw Learn Syst , 28 ( 10 ): 2434 - 2445 . doi: 10.1109/TNNLS.2016.2609500 http://doi.org/10.1109/TNNLS.2016.2609500

Li JQ , Wang QL , Su YX , et al. , 2021 . Robust distributed model predictive consensus of discrete-time multi-agent systems: a self-triggered approach . Front Inform Technol Electron Eng , 22 ( 8 ): 1068 - 1079 . doi: 10.1631/FITEE.2000182 http://doi.org/10.1631/FITEE.2000182

Liu DR , Xue S , Zhao B , et al. , 2021 . Adaptive dynamic programming for control: a survey and recent advances . IEEE Trans Syst Man Cybern Syst , 51 ( 1 ): 142 - 160 . doi: 10.1109/TSMC.2020.3042876 http://doi.org/10.1109/TSMC.2020.3042876

Ma HJ , Yang GH , 2016 . Adaptive fault tolerant control of cooperative heterogeneous systems with actuator faults and unreliable interconnections . IEEE Trans Autom Contr , 61 ( 11 ): 3240 - 3255 . doi: 10.1109/TAC.2015.2507864 http://doi.org/10.1109/TAC.2015.2507864

Qin JH , Li M , Shi Y , et al. , 2019 . Optimal synchronization control of multiagent systems with input saturation via off-policy reinforcement learning . IEEE Trans Neur Netw Learn Syst , 30 ( 1 ): 85 - 96 . doi: 10.1109/TNNLS.2018.2832025 http://doi.org/10.1109/TNNLS.2018.2832025

Rehák B , Lynnyk V , 2021 . Leader-following synchronization of a multi-agent system with heterogeneous delays . Front Inform Technol Electron Eng , 22 ( 1 ): 97 - 106 . doi: 10.1631/FITEE.2000207 http://doi.org/10.1631/FITEE.2000207

Thunberg J , Song W , Monitijano E , et al. , 2014 . Distributed attitude synchronization control of multi-agent systems with switching topologies . Automatica , 50 ( 3 ): 832 - 840 . doi: 10.1016/j.automatica.2014.02.002 http://doi.org/10.1016/j.automatica.2014.02.002

Vamvoudakis KG , Lewis FL , Hudas GR , 2012 . Multi-agent differential graphical games: online adaptive learning solution for synchronization with optimality . Automatica , 48 ( 8 ): 1598 - 1611 . doi: 10.1016/j.automatica.2012.05.074 http://doi.org/10.1016/j.automatica.2012.05.074

Vrabie D , Lewis F , 2011 . Adaptive dynamic programming for online solution of a zero-sum differential game . J Contr Theory Appl , 9 ( 3 ): 353 - 360 . doi: 10.1007/s11768-011-0166-4 http://doi.org/10.1007/s11768-011-0166-4

Wang FY , Zhang HG , Liu DR , 2009 . Adaptive dynamic programming: an introduction . IEEE Comput Intell Mag , 4 ( 2 ): 39 - 47 . doi: 10.1109/MCI.2009.932261 http://doi.org/10.1109/MCI.2009.932261

Wei QL , Liu DR , 2014 . Adaptive dynamic programming for optimal tracking control of unknown nonlinear systems with application to coal gasification . IEEE Trans Autom Sci Eng , 11 ( 4 ): 1020 - 1036 . doi: 10.1109/TASE.2013.2284545 http://doi.org/10.1109/TASE.2013.2284545

Wei QL , Wang FY , Liu DR , et al. , 2014 . Finite-approximation-error-based discrete-time iterative adaptive dynamic programming . IEEE Trans Cybern , 44 ( 12 ): 2820 - 2833 . doi: 10.1109/TCYB.2014.2354377 http://doi.org/10.1109/TCYB.2014.2354377

Wei QL , Liu DR , Lewis FL , 2015 . Optimal distributed synchronization control for continuous-time heterogeneous multi-agent differential graphical games . Inform Sci , 317 : 96 - 113 . doi: 10.1016/j.ins.2015.04.044 http://doi.org/10.1016/j.ins.2015.04.044

Wei QL , Liu DR , Lin HQ , 2016 . Value iteration adaptive dynamic programming for optimal control of discrete-time nonlinear systems . IEEE Trans Cybern , 46 ( 3 ): 840 - 853 . doi: 10.1109/TCYB.2015.2492242 http://doi.org/10.1109/TCYB.2015.2492242

Wei QL , Lewis FL , Sun QY , et al. , 2017 . Discrete-time deterministic Q -learning: a novel convergence analysis . IEEE Trans Cybern , 47 ( 5 ): 1224 - 1237 . doi: 10.1109/TCYB.2016.2542923 http://doi.org/10.1109/TCYB.2016.2542923

Wei QL , Lewis FL , Liu DR , et al. , 2018 . Discrete-time local value iteration adaptive dynamic programming: convergence analysis . IEEE Trans Syst Man Cybern Syst , 48 ( 6 ): 875 - 891 . doi: 10.1109/TSMC.2016.2623766 http://doi.org/10.1109/TSMC.2016.2623766

Wei QL , Li HY , Wang FY , 2020 . Parallel control for continuous-time linear systems: a case study . IEEE/ CAA J Autom Sin , 7 ( 4 ): 919 - 928 . doi: 10.1109/JAS.2020.1003216 http://doi.org/10.1109/JAS.2020.1003216

Wei QL , Wang X , Zhong XN , et al. , 2021 . Consensus control of leader-following multi-agent systems in directed topology with heterogeneous disturbances . IEEE/CAA J Autom Sin , 8 ( 2 ): 423 - 431 . doi: 10.1109/JAS.2021.1003838 http://doi.org/10.1109/JAS.2021.1003838

Wieland P , Sepulchre R , Allgöwer F , 2011 . An internal model principle is necessary and sufficient for linear output synchronization . Automatica , 47 ( 5 ): 1068 - 1074 . doi: 10.1016/j.automatica.2011.01.081 http://doi.org/10.1016/j.automatica.2011.01.081

Yang JY , Xi F , Ma J , 2019 . Model-based edge-event-triggered containment control under directed topologies . IEEE Trans Cybern , 49 ( 7 ): 2556 - 2567 . doi: 10.1109/TCYB.2018.2828645 http://doi.org/10.1109/TCYB.2018.2828645

Yang N , Xiao JW , Xiao L , et al. , 2019 . Non-zero sum differential graphical game: cluster synchronisation for multi-agents with partially unknown dynamics . Int J Contr , 92 ( 10 ): 2408 - 2419 . doi: 10.1080/00207179.2018.1441550 http://doi.org/10.1080/00207179.2018.1441550

Zhang HG , Zhang JL , Yang GH , et al. , 2015 . Leader-based optimal coordination control for the consensus problem of multiagent differential games via fuzzy adaptive dynamic programming . IEEE Trans Fuzzy Syst , 23 ( 1 ): 152 - 163 . doi: 10.1109/TFUZZ.2014.2310238 http://doi.org/10.1109/TFUZZ.2014.2310238

Zhang KQ , Yang ZR , Başar T , 2021 . Decentralized multi-agent reinforcement learning with networked agents: recent advances . Front Inform Technol Electron Eng , 22 ( 6 ): 802 - 814 . doi: 10.1631/FITEE.1900661 http://doi.org/10.1631/FITEE.1900661

Zhang LD , Wang B , Liu ZX , et al. , 2019 . Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method . Front Inform Technol Electron Eng , 20 ( 4 ): 525 - 537 . doi: 10.1631/FITEE.1800571 http://doi.org/10.1631/FITEE.1800571

Zhao DY , Zhu QM , Li N , et al. , 2014 . Synchronized control with neuro-agents for leader-follower based multiple robotic manipulators . Neurocomputing , 124 : 149 - 161 . doi: 10.1016/j.neucom.2013.07.016 http://doi.org/10.1016/j.neucom.2013.07.016

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Minimax Q-learning design for H_∞ control of linear discrete-time systems

Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Towards resilient average consensus in multi-agent systems: a detection and compensation approach

Event-triggered distributed optimization for model-free multi-agent systems

Bipartite asynchronous impulsive tracking consensus for multi-agent systems

Related Author

Lele XI

Wenzhong ZHA

Zhihong PENG

Xinxing LI

Dong YUE

Fanming HUANG

Yang YANG

Jingpei WANG

Related Institution

School of Automation, Beijing Institute of Technology

Peng Cheng Laboratory

Information Science Academy, China Electronics Technology Group Corporation

Advanced Technology Institute for Carbon Neutrality, Nanjing University of Posts and Telecommunications

College of Automation & College of Artificial Intelligence, Nanjing University of Posts and Telecommunications

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰