Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Yang YANG; Fanming HUANG; Dong YUE

doi:10.1631/FITEE.2300532

Your Location：

Home >

Browse articles >

Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Regular Papers | Updated：2025-04-03

- Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems
  Enhanced Publication
- 基于强化学习的非严格反馈离散时间多智能体系统隐私保护一致性跟踪控制
- Frontiers of Information Technology & Electronic Engineering Vol. 26, Issue 3, Pages: 456-471(2025)
- Affiliations：
  
  1.College of Automation & College of Artificial Intelligence, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
  2.Advanced Technology Institute for Carbon Neutrality, Nanjing University of Posts and Telecommunications, Nanjing 210023, China
- Author bio：
  
  E-mail: yyang@njupt.edu.cn
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2300532
  CLC： TP13
- Received：06 August 2023，
  
  Revised：2023-11-22，
  
  Published：2025-03
- Accepted：
Scan QR Code
Yang YANG, Fanming HUANG, Dong YUE. Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems[J]. Frontiers of Information Technology & Electronic Engineering, 2025, 26(3): 456-471.
DOI：

Yang YANG, Fanming HUANG, Dong YUE. Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems[J]. Frontiers of Information Technology & Electronic Engineering, 2025, 26(3): 456-471. DOI： 10.1631/FITEE.2300532.

摘要

本文研究了一类非严格反馈离散时间多智能体系统的隐私保护一致性跟踪问题。为减轻明文加密和解密之间的误差影响，开发一种改进的Liu加密系统，以确保明文信息恢复良好。采用强化学习技术补偿未知动态和真实信号与解密信号之间的误差。采用反步法和图论知识，设计基于强化学习的隐私保护一致性跟踪控制策略。借助李雅普诺夫稳定性理论，证明多智能体系统的一致跟踪误差和所有信号最终有界。最后，通过仿真实例验证设计控制策略的有效性。

Abstract

This paper investigates a privacy-preserving consensus tracking problem for a class of nonstrict-feedback discrete-time multi-agent systems (MASs). An improved Liu cryptosystem is developed to alleviate the errors between encryption and decryption on the plaintext

which ensures satisfactory recovery of the plaintext information. A reinforcement learning (RL) technique is then employed to compensate for unknown dynamics and errors between true signals and decrypted ones. Based on the backstepping and graph theory

an RL-based privacy-preserving consensus tracking control strategy is further designed. By virtue of graph theory and Lyapunov stability theory

it is shown that the consensus tracking errors and all signals in the MAS are ultimately bounded. Finally

simulation examples are presented for verification of the effectiveness of the control strategy.

关键词

Keywords

references

Bai WW , Zhang B , Zhou Q , et al. , 2020a . Multigradient recursive reinforcement learning NN control for affine nonlinear systems with unmodeled dynamics . Int J Robust Nonl Contr , 30 ( 4 ): 1643 - 1663 . https://doi.org/10.1002/rnc.4843 https://doi.org/10.1002/rnc.4843

Bai WW , Li TS , Tong SC , 2020b . NN reinforcement learning adaptive control for a class of nonstrict-feedback discrete-time systems . IEEE Trans Cybern , 50 ( 11 ): 4573 - 4584 . https://doi.org/10.1109/TCYB.2020.2963849 https://doi.org/10.1109/TCYB.2020.2963849

Chen W , Liu L , Liu GP , 2023 . Privacy-preserving distributed economic dispatch of microgrids: a dynamic quantization-based consensus scheme with homomorphic encryption . IEEE Trans Smart Grid , 14 ( 1 ): 701 - 713 . https://doi.org/10.1109/TSG.2022.3189665 https://doi.org/10.1109/TSG.2022.3189665

Ding L , Li S , Gao HB , et al. , 2020 . Adaptive partial reinforcement learning neural network-based tracking control for wheeled mobile robotic systems . IEEE Trans Syst Man Cybern Syst , 50 ( 7 ): 2512 - 2523 . https://doi.org/10.1109/TSMC.2018.2819191 https://doi.org/10.1109/TSMC.2018.2819191

Ding L , Li S , Gao HB , et al. , 2021 . Adaptive neural network-based finite-time online optimal tracking control of the nonlinear system with dead zone . IEEE Trans Cybern , 51 ( 1 ): 382 - 392 . https://doi.org/10.1109/TCYB.2019.2939424 https://doi.org/10.1109/TCYB.2019.2939424

Elgamal T , 1985 . A public key cryptosystem and a signature scheme based on discrete logarithms . IEEE Trans Inform Theory , 31 ( 4 ): 469 - 472 . https://doi.org/10.1109/TIT.1985.1057074 https://doi.org/10.1109/TIT.1985.1057074

Fang WT , Zamani M , Chen ZY , 2021 . Secure and privacy preserving consensus for second-order systems based on Paillier encryption . Syst Contr Lett , 148 : 104869 . https://doi.org/10.1016/j.sysconle.2020.104869 https://doi.org/10.1016/j.sysconle.2020.104869

Gao C , Wang ZD , He X , et al. , 2021 . Encryption–decryption-based consensus control for multi-agent systems: handling actuator faults . Automatica , 134 : 109908 . https://doi.org/10.1016/j.automatica.2021.109908 https://doi.org/10.1016/j.automatica.2021.109908

Gao L , Deng SJ , Ren W , 2019 . Differentially private consensus with an event-triggered mechanism . IEEE Trans Contr Netw Syst , 6 ( 1 ): 60 - 71 . https://doi.org/10.1109/TCNS.2018.2795703 https://doi.org/10.1109/TCNS.2018.2795703

Ge SS , Li GY , Lee TH , 2003 . Adaptive NN control for a class of strict-feedback discrete-time nonlinear systems . Automatica , 39 ( 5 ): 807 - 819 . https://doi.org/10.1016/S0005-1098(03)00032-3 https://doi.org/10.1016/S0005-1098(03)00032-3

Ge XH , Xiao SY , Han QL , et al. , 2022 . Dynamic event-triggered scheduling and platooning control co-design for automated vehicles over vehicular ad-hoc networks . IEEE/CAA J Autom Sin , 9 ( 1 ): 31 - 46 . https://doi.org/10.1109/JAS.2021.1004060 https://doi.org/10.1109/JAS.2021.1004060

Ge XH , Han QL , Wu Q , et al. , 2023 . Resilient and safe platooning control of connected automated vehicles against intermittent denial-of-service attacks . IEEE/CAA J Autom Sin , 10 ( 5 ): 1234 - 1251 . https://doi.org/10.1109/JAS.2022.105845 https://doi.org/10.1109/JAS.2022.105845

Ge XH , Han QL , Zhang XM , et al. , 2024 . Communication resource-efficient vehicle platooning control with various spacing policies . IEEE/CAA J Autom Sin , 11 ( 2 ): 362 - 376 . https://doi.org/10.1109/JAS.2023.123507 https://doi.org/10.1109/JAS.2023.123507

He WL , Xu WY , Ge XH , et al. , 2022 . Secure control of multiagent systems against malicious attacks: a brief survey . IEEE Trans Ind Inform , 18 ( 6 ): 3595 - 3608 . https://doi.org/10.1109/TII.2021.3126644 https://doi.org/10.1109/TII.2021.3126644

Ju YM , Ding DR , He X , et al. , 2022 . Consensus control of multi-agent systems using fault-estimation-in-the-loop: dynamic event-triggered case . IEEE/CAA J Autom Sin , 9 ( 8 ): 1440 - 1451 . https://doi.org/10.1109/JAS.2021.1004386 https://doi.org/10.1109/JAS.2021.1004386

Kishida M , 2019 . Encrypted control system with quantiser . IET Contr Theory Appl , 13 ( 1 ): 146 - 151 . https://doi.org/10.1049/iet-cta.2018.5764 https://doi.org/10.1049/iet-cta.2018.5764

Li HY , Wu Y , Chen M , 2021 . Adaptive fault-tolerant tracking control for discrete-time multiagent systems via reinforcement learning algorithm . IEEE Trans Cybern , 51 ( 3 ): 1163 - 1174 . https://doi.org/10.1109/TCYB.2020.2982168 https://doi.org/10.1109/TCYB.2020.2982168

Li JN , Yuan L , Chai TY , et al. , 2023 . Consensus of nonlinear multiagent systems with uncertainties using reinforcement learning based sliding mode control . IEEE Trans Circ Syst I Regular Papers , 70 ( 1 ): 424 - 434 . https://doi.org/10.1109/TCSI.2022.3206102 https://doi.org/10.1109/TCSI.2022.3206102

Li P , Hu JP , Qiu L , et al. , 2022 . A distributed economic dispatch strategy for power-water networks . IEEE Trans Contr Netw Syst , 9 ( 1 ): 356 - 366 . https://doi.org/10.1109/TCNS.2021.3104103 https://doi.org/10.1109/TCNS.2021.3104103

Li YM , Min X , Tong SC , 2020a . Adaptive fuzzy inverse optimal control for uncertain strict-feedback nonlinear systems . IEEE Trans Fuzzy Syst , 28 ( 10 ): 2363 - 2374 . https://doi.org/10.1109/TFUZZ.2019.2935693 https://doi.org/10.1109/TFUZZ.2019.2935693

Li YM , Shao XF , Tong SC , 2020b . Adaptive fuzzy prescribed performance control of nontriangular structure nonlinear systems . IEEE Trans Fuzzy Syst , 28 ( 10 ): 2416 - 2426 . https://doi.org/10.1109/TFUZZ.2019.2937046 https://doi.org/10.1109/TFUZZ.2019.2937046

Li YM , Liu YJ , Tong SC , 2022 . Observer-based neuro-adaptive optimized control of strict-feedback nonlinear systems with state constraints . IEEE Trans Neur Netw Learn Syst , 33 ( 7 ): 3131 - 3145 . https://doi.org/10.1109/TNNLS.2021.3051030 https://doi.org/10.1109/TNNLS.2021.3051030

Liang CD , Ge MF , Xu JZ , et al. , 2022 . Secure and privacy-preserving formation control for networked marine surface vehicles with sampled-data interactions . IEEE Trans Veh Technol , 71 ( 2 ): 1307 - 1318 . https://doi.org/10.1109/TVT.2021.3133902 https://doi.org/10.1109/TVT.2021.3133902

Lin XZ , Chen CC , Li SH , 2022 . Finite-time output feedback stabilization for a class of output-constrained planar switched systems . IEEE Trans Circ Syst II Express Briefs , 69 ( 1 ): 164 - 168 . https://doi.org/10.1109/TCSII.2021.3077603 https://doi.org/10.1109/TCSII.2021.3077603

Liu DX , 2013 . Homomorphic Encryption for Database Querying . US Patent 20150295716 .

Liu L , Ding SH , Yu XH , 2021 . Second-order sliding mode control design subject to an asymmetric output constraint . IEEE Trans Circ Syst II Express Briefs , 68 ( 4 ): 1278 - 1282 . https://doi.org/10.1109/TCSII.2020.3021715 https://doi.org/10.1109/TCSII.2020.3021715

Liu L , Cui YJ , Liu YJ , et al. , 2022 . Adaptive event-triggered output feedback control for nonlinear switched systems based on full state constraints . IEEE Trans Circ Syst II Express Briefs , 69 ( 9 ): 3779 - 3783 . https://doi.org/10.1109/TCSII.2022.3173679 https://doi.org/10.1109/TCSII.2022.3173679

Ning BD , Han QL , Zuo ZY , et al. , 2023 . Fixed-time and prescribed-time consensus control of multiagent systems and its applications: a survey of recent trends and methodologies . IEEE Trans Ind Inform , 19 ( 2 ): 1121 - 1135 . https://doi.org/10.1109/TII.2022.3201589 https://doi.org/10.1109/TII.2022.3201589

Nozari E , Tallapragada P , Cortés J , 2017 . Differentially private average consensus: obstructions, trade-offs, and optimal algorithm design . Automatica , 81 : 221 - 231 . https://doi.org/10.1016/j.automatica.2017.03.016 https://doi.org/10.1016/j.automatica.2017.03.016

Peng ZH , Wang J , Wang D , et al. , 2021 . An overview of recent advances in coordinated control of multiple autonomous surface vehicles . IEEE Trans Ind Inform , 17 ( 2 ): 732 - 745 . https://doi.org/10.1109/TII.2020.3004343 https://doi.org/10.1109/TII.2020.3004343

Qi XJ , Liu WH , Lu Y , 2023 . Event-triggered-based fuzzy adaptive tracking control for nonstrict-feedback asymmetric state constrained systems . Fuzzy Sets Syst , 470 : 108642 . https://doi.org/10.1016/j.fss.2023.108642 https://doi.org/10.1016/j.fss.2023.108642

Reddy SS , Sinha S , Zhang W , 2023 . Design and analysis of RSA and Paillier homomorphic cryptosystems using PSO-based evolutionary computation . IEEE Trans Comput , 72 ( 7 ): 1886 - 1900 . https://doi.org/10.1109/TC.2023.3234213 https://doi.org/10.1109/TC.2023.3234213

Rivest RL , Shamir A , Adleman L , 1978 . A method for obtaining digital signatures and public-key cryptosystems . Commun ACM , 21 ( 2 ): 120 - 126 . https://doi.org/10.1145/359340.359342 https://doi.org/10.1145/359340.359342

Ruan MH , Gao H , Wang YQ , 2019 . Secure and privacy-preserving consensus . IEEE Trans Autom Contr , 64 ( 10 ): 4035 - 4049 . https://doi.org/10.1109/TAC.2019.2890887 https://doi.org/10.1109/TAC.2019.2890887

Sakthivel R , Sakthivel R , Kaviarasan B , et al. , 2019 . Finite-time leaderless consensus of uncertain multi-agent systems against time-varying actuator faults . Neurocomputing , 325 : 159 - 171 . https://doi.org/10.1016/j.neucom.2018.10.020 https://doi.org/10.1016/j.neucom.2018.10.020

Shahvali M , Naghibi-Sistani MB , Askari J , 2018 . Adaptive output-feedback bipartite consensus for nonstrict-feedback nonlinear multi-agent systems: a finite-time approach . Neurocomputing , 318 : 7 - 17 . https://doi.org/10.1016/j.neucom.2018.07.039 https://doi.org/10.1016/j.neucom.2018.07.039

Sun JL , Yi JQ , Pu ZQ , 2022 . Fixed-time adaptive fuzzy control for uncertain nonstrict-feedback systems with time-varying constraints and input saturations . IEEE Trans Fuzzy Syst , 30 ( 4 ): 1114 - 1128 . https://doi.org/10.1109/TFUZZ.2021.3052610 https://doi.org/10.1109/TFUZZ.2021.3052610

Sun KK , Qiu JB , Karimi HR , et al. , 2021 . A novel finite-time control for nonstrict feedback saturated nonlinear systems with tracking error constraint . IEEE Trans Syst Man Cybern Syst , 51 ( 6 ): 3968 - 3979 . https://doi.org/10.1109/TSMC.2019.2958072 https://doi.org/10.1109/TSMC.2019.2958072

Tong SC , Li YM , Sui S , 2016 . Adaptive fuzzy tracking control design for SISO uncertain nonstrict feedback non-linear systems . IEEE Trans Fuzzy Syst , 24 ( 6 ): 1441 - 1454 . https://doi.org/10.1109/TFUZZ.2016.2540058 https://doi.org/10.1109/TFUZZ.2016.2540058

Tong SC , Sun KK , Sui S , 2018 . Observer-based adaptive fuzzy decentralized optimal control design for strict-feedback nonlinear large-scale systems . IEEE Trans Fuzzy Syst , 26 ( 2 ): 569 - 584 . https://doi.org/10.1109/TFUZZ.2017.2686373 https://doi.org/10.1109/TFUZZ.2017.2686373

Wang AJ , Liu WP , Dong T , et al. , 2022 . DisEHPPC: enabling heterogeneous privacy-preserving consensus-based scheme for economic dispatch in smart grids . IEEE Trans Cybern , 52 ( 6 ): 5124 - 5135 . https://doi.org/10.1109/TCYB.2020.3027572 https://doi.org/10.1109/TCYB.2020.3027572

Wang H , Li M , 2022 . Model-free reinforcement learning for fully cooperative consensus problem of nonlinear multiagent systems . IEEE Trans Neur Netw Learn Syst , 33 ( 4 ): 1482 - 1491 . https://doi.org/10.1109/TNNLS.2020.3042508 https://doi.org/10.1109/TNNLS.2020.3042508

Wang SB , 2022 . Asymptotic tracking control for nonaffine systems with disturbances . IEEE Trans Circ Syst II Express Briefs , 69 ( 2 ): 479 - 483 . https://doi.org/10.1109/TCSII.2021.3080524 https://doi.org/10.1109/TCSII.2021.3080524

Wang YQ , 2019 . Privacy-preserving average consensus via state decomposition . IEEE Trans Autom Contr , 64 ( 11 ): 4711 - 4716 . https://doi.org/10.1109/TAC.2019.2902731 https://doi.org/10.1109/TAC.2019.2902731

Wang YQ , Lu JQ , Zheng WX , et al. , 2021 . Privacy-preserving consensus for multi-agent systems via node decomposition strategy . IEEE Trans Circ Syst I Regular Papers , 68 ( 8 ): 3474 - 3484 . https://doi.org/10.1109/TCSI.2021.3081372 https://doi.org/10.1109/TCSI.2021.3081372

Wen GX , Li B , 2022 . Optimized leader-follower consensus control using reinforcement learning for a class of second-order nonlinear multiagent systems . IEEE Trans Syst Man Cybern Syst , 52 ( 9 ): 5546 - 5555 . https://doi.org/10.1109/TSMC.2021.3130070 https://doi.org/10.1109/TSMC.2021.3130070

Xie ML , Ding DR , Ge XH , et al. , 2022 . Distributed platooning control of automated vehicles subject to replay attacks based on proportional integral observers . IEEE/CAA J Autom Sin , 11 ( 9 ): 1954 - 1966 . https://doi.org/10.1109/JAS.2022.105941 https://doi.org/10.1109/JAS.2022.105941

Xu HY , Ni YH , Liu ZX , et al. , 2021 . Privacy-preserving leader-following consensus via node-augment mechanism . IEEE Trans Circ Syst II Express Briefs , 68 ( 6 ): 2117 - 2121 . https://doi.org/10.1109/TCSII.2020.3047850 https://doi.org/10.1109/TCSII.2020.3047850

Yan YM , Chen ZY , Varadharajan V , et al. , 2021 . Distributed consensus-based economic dispatch in power grids using the Paillier cryptosystem . IEEE Trans Smart Grid , 12 ( 4 ): 3493 - 3502 . https://doi.org/10.1109/TSG.2021.3063712 https://doi.org/10.1109/TSG.2021.3063712

Yang XD , Zhang H , Wang ZP , 2022 . Data-based optimal consensus control for multi-agent systems with policy gradient reinforcement learning . IEEE Trans Neur Netw Learn Syst , 33 ( 8 ): 3872 - 3883 . https://doi.org/10.1109/TNNLS.2021.3054685 https://doi.org/10.1109/TNNLS.2021.3054685

Yang ZW , Yu LY , Liu YR , et al. , 2022 . Event-triggered privacy-preserving bipartite consensus for multi-agent systems based on encryption . Neurocomputing , 503 : 162 - 172 . https://doi.org/10.1016/j.neucom.2022.06.074 https://doi.org/10.1016/j.neucom.2022.06.074

Yin TJ , Lv YZ , Yu WW , 2020 . Accurate privacy preserving average consensus . IEEE Trans Circ Syst II Express Briefs , 67 ( 4 ): 690 - 694 . https://doi.org/10.1109/TCSII.2019.2918709 https://doi.org/10.1109/TCSII.2019.2918709

Yu T , Ma L , Zhang HW , 2019 . Prescribed performance for bipartite tracking control of nonlinear multi-agent systems with hysteresis input uncertainties . IEEE Trans Cybern , 49 ( 4 ): 1327 - 1338 . https://doi.org/10.1109/TCYB.2018.2800297 https://doi.org/10.1109/TCYB.2018.2800297

Zhang HG , Jiang H , Luo YH , et al. , 2017 . Data-driven optimal consensus control for discrete-time multi-agent systems with unknown dynamics using reinforcement learning method . IEEE Trans Ind Electron , 64 ( 5 ): 4091 - 4100 . https://doi.org/10.1109/TIE.2016.2542134 https://doi.org/10.1109/TIE.2016.2542134

Zhang JX , Li KW , Li YM , 2021 . Output-feedback based simplified optimized backstepping control for strict-feedback systems with input and state constraints . IEEE/CAA J Autom Sin , 8 ( 6 ): 1119 - 1132 . https://doi.org/10.1109/JAS.2021.1004018 https://doi.org/10.1109/JAS.2021.1004018

Zhang P , Huang T , Sun XQ , et al. , 2023 . Privacy-preserving and outsourced multi-party K-means clustering based on multi-key fully homomorphic encryption . IEEE Trans Dependab Secure Comput , 20 ( 3 ): 2348 - 2359 . https://doi.org/10.1109/TDSC.2022.3181667 https://doi.org/10.1109/TDSC.2022.3181667

Zhang XM , Han QL , Ge XH , et al. , 2023 . Sampled-data control systems with non-uniform sampling: a survey of methods and trends . Annu Rev Contr , 55 : 70 - 91 . https://doi.org/10.1016/j.arcontrol.2023.03.004 https://doi.org/10.1016/j.arcontrol.2023.03.004

Zhou Q , Li HY , Wang LJ , et al. , 2018 . Prescribed performance observer-based adaptive fuzzy control for nonstrict-feedback stochastic nonlinear systems . IEEE Trans Syst Man Cybern Syst , 48 ( 10 ): 1747 - 1758 . https://doi.org/10.1109/TSMC.2017.2738155 https://doi.org/10.1109/TSMC.2017.2738155

Zuo XJ , Li LX , Peng HP , et al. , 2021 . Privacy-preserving multidimensional data aggregation scheme without trusted authority in smart grid . IEEE Syst J , 15 ( 1 ): 395 - 406 . https://doi.org/10.1109/JSYST.2020.2994363 https://doi.org/10.1109/JSYST.2020.2994363

Views

205

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Privacy-preserving bipartite consensus with cooperative–competitive interactions via a node decomposition strategy

Decentralized multi-agent reinforcement learning with networked agents: recent advances

Jiu fusion artificial intelligence (JFA): a two-stage reinforcement learning model with hierarchical neural networks and human knowledge for Tibetan Jiu chess

Efficient learning of robust multigait quadruped locomotion for minimizing the cost of transport

Efficient privacy-preserving scheme for secure neural network inference

Related Author

Licheng WANG

Yongling CHEN

Shuai LIU

Kaiqing ZHANG

Zhuoran YANG

Tamer BAŞAR

Xiali LI

Xiaoyu FAN

Related Institution

College of Science, University of Shanghai for Science and Technology

College of Automation Engineering, Shanghai University of Electric Power

Department of Operations Research and Financial Engineering, Princeton University

Coordinated Science Laboratory, University of Illinois at Urbana-Champaign

School of Information Science and Technology, Xizang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰