Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method

Li-dong ZHANG; Ban WANG; Zhi-xiang LIU; You-min ZHANG; Jian-liang AI

doi:10.1631/FITEE.1800571

Your Location：

Home >

Browse articles >

Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method

Regular Papers | Updated：2022-05-19

- Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method
- 四旋翼无人机在博弈中的运动规划研究：一种基于仿真的投影策略迭代方法
- Frontiers of Information Technology & Electronic Engineering Vol. 20, Issue 4, Pages: 525-537(2019)
- Affiliations：
  
  Department of Aeronautics and Astronautics, Fudan University, Shanghai 200433, China
  Department of Mechanical, Industrial and Aerospace Engineering, Concordia University, Quebec H3G1M8, Canada
- Author bio：
  
  [ "Li-dong ZHANG, E-mail: lidongzhang13@fudan.edu.cn" ]
  You-min ZHANG, E-mail: ymzhang@encs.concordia.ca
- Funds：
- DOI：10.1631/FITEE.1800571
  CLC： TP242
- Received：15 September 2018，
  
  Revised：2019-;4-28，
  
  Published：2019-04
- Accepted：
Scan QR Code
Li-dong ZHANG, Ban WANG, Zhi-xiang LIU, et al. Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(4): 525-537.
DOI：

Li-dong ZHANG, Ban WANG, Zhi-xiang LIU, et al. Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(4): 525-537. DOI： 10.1631/FITEE.1800571.

摘要

数十年来，如何实现在复杂环境中对序列决策问题做出有效合理的决策始终是一个困扰各领域研究者的难题。该决策问题包含状态转移动力学模型、随机因素引入的不确定性、长远决策前沿优化问题以及其他许多难题，包括维数灾难在内的诸多困难使解决这一决策问题的有效方法仍待进一步研究探索。目前，随着增强学习领域不断开发出先进算法，为尝试解决复杂环境中的序列决策问题提供了有潜力的解决方案，并可在实际应用环境中获得较高决策性能。本文提出三维空间中速度可变的一对一四旋翼无人机博弈研究平台，以及一种基于投影策略迭代方法的近似动态规划方法，以学习四旋翼博弈过程中效用函数并生成改进的四旋翼运动决策策略。此外，采用基于仿真的方法，消除维数灾难束缚。仿真结果表明，所提决策方法可在博弈对抗中高效生成有效运动策略，并在与对方四旋翼无人机对抗中能获取并保持有利态势。在肯高迪亚大学网络自动车辆（NAV）实验室进行的飞行实验进一步验证了所提决策方法在实时环境中的决策性能。

Abstract

Making rational decisions for sequential decision problems in complex environments has been challenging researchers in various fields for decades. Such problems consist of state transition dynamics

stochastic uncertainties

long-term utilities

and other factors that assemble high barriers including the curse of dimensionality. Recently

the state-of-the-art algorithms in reinforcement learning studies have been developed

providing a strong potential to efficiently break the barriers and make it possible to deal with complex and practical decision problems with decent performance. We propose a formulation of a velocity varying one-on-one quadrotor robot game problem in the three-dimensional space and an approximate dynamic programming approach using a projected policy iteration method for learning the utilities of game states and improving motion policies. In addition

a simulation-based iterative scheme is employed to overcome the curse of dimensionality. Simulation results demonstrate that the proposed decision strategy can generate effective and efficient motion policies that can contend with the opponent quadrotor and gather advantaged status during the game. Flight experiments

which are conducted in the Networked Autonomous Vehicles (NAV) Lab at the Concordia University

have further validated the performance of the proposed decision strategy in the real-time environment.

关键词

Keywords

references

BW Ballard . . The *-minimax search procedure for trees containing chance nodes . . Artif Intell , , 1983 . . 21 ( ( 3 ): ): 327 - - 350 . . DOI: 10.1016/S0004-3702(83)80015-0 http://doi.org/10.1016/S0004-3702(83)80015-0 . .

R Bellman . . On the theory of dynamic programming . . Proc Nat Acad Sci , , 1952 . . 38 ( ( 8 ): ): 716 - - 719 . . DOI: 10.1073/pnas.38.8.716 http://doi.org/10.1073/pnas.38.8.716 . .

DP Bertsekas . . Dynamic Programming and Optimal Control. Athena Scientific, Belmont . . Belmont, Massachusetts, USA , , 1971 . . .

DP Bertsekas . . Dynamic Programming and Optimal Control (3 $$.{\rm rd}$$ Ed.) Athena Scientific, Belmont . . Massachusetts, USA. , , 2007 . . .

DP Bertsekas . . Temporal difference methods for general projected equations . . IEEE Trans Autom Contr , , 2011 . . 56 ( ( 9 ): ): 2128 - - 2139 . . DOI: 10.1109/TAC.2011.2115290 http://doi.org/10.1109/TAC.2011.2115290 . .

DP Bertsekas . . Incremental gradient, subgradient, and proximal methods for convex optimization: a survey. In: Suvrit Sra SN, Wright SJ (Eds.)Optimization for Machine Learning , , : : Massachusetts, USA MIT Press , , 2012 . . .

DP Bertsekas . . Lambda-policy iteration: a review and a new implementation . . 2015 . . https://arxiv.org/abs/1507.01029 https://arxiv.org/abs/1507.01029 , , . .

DP Bertsekas , , , JN Tsitsiklis . . Gradient convergence in gradient methods with errors . . SIAM J Optim , , 2000 . . 10 ( ( 3 ): ): 627 - - 642 . . DOI: 10.1137/S1052623497331063 http://doi.org/10.1137/S1052623497331063 . .

L Buşoniu , , , D Ernst , , , B de Schutter , , , 等 . . Online least-squares policy iteration for reinforcement learning control . . Proc American Control Conf , , 2010 . . p.486 - - 491 . . DOI: 10.1109/ACC.2010.5530856 http://doi.org/10.1109/ACC.2010.5530856 . .

Y Efroni , , , G Dalal , , , B Scherrer , , , 等 . . Beyond the one step greedy approach in reinforcement learning . . 2018a . . https://arxiv.org/abs/1802.03654 https://arxiv.org/abs/1802.03654 , , . .

Y Efroni , , , G Dalal , , , B Scherrer , , , 等 . . Multiple-step greedy policies in online and approximate reinforcement learning . . 2018b . . https://arxiv.org/abs/1805.07956 https://arxiv.org/abs/1805.07956 , , . .

J Fang , , , LM Zhang , , , W Fang , , , 等 . . Approximate dynamic programming for CGF air combat maneuvering decision . . 2 nd IEEE Int Conf on Computer and Communications , , 2016 . . p.1386 - - 1390 . . DOI: 10.1109/CompComm.2016.7924931 http://doi.org/10.1109/CompComm.2016.7924931 . .

KA Ghamry , , , YQ Dong , , , MA Kamel , , , 等 . . Real-time autonomous take-off, tracking and landing of UAV on a moving UGV platform . . 24 th Mediterranean Conf on Control and Automation , , 2016 . . p.1236 - - 1241 . . DOI: 10.1109/MED.2016.7535886 http://doi.org/10.1109/MED.2016.7535886 . .

T Hastie , , , R Tibshirani , , , J Friedman . . The Elements of Statistical Learning , , : : New York, USA Springer , , 2001 . . .

T Hauk , , , M Buro , , , J Schaeffer . . Rediscovering *-minimax search . . Int Conf on Computers and Games , , 2004 . . p.35 - - 50 . . DOI: 10.1007/11674399_3 http://doi.org/10.1007/11674399_3 . .

ZX Liu , , , YM Zhang , , , X Yu , , , 等 . . Unmanned surface vehicles: an overview of developments and challenges . . Ann Rev Contr , , 2016 . . 41 71 - - 93 . . DOI: 10.1016/j.arcontrol.2016.04.018 http://doi.org/10.1016/j.arcontrol.2016.04.018 . .

YF Ma , , , XL Ma , , , X Song . . A case study on air combat decision using approximated dynamic programming . . Math Probl Eng , , 2014 . . 2014 183401 DOI: 10.1155/2014/183401 http://doi.org/10.1155/2014/183401 . .

JS McGrew . . Real-Time Maneuvering Decisions for Autonomous Air Combat. MS Thesis, Massachusetts Institute of Technology . . Massachusetts, USA. , , 2008 . . .

JS McGrew , , , JP How , , , B Williams , , , 等 . . Air-combat strategy using approximate dynamic programming . . J Guid Contr Dynam , , 2010 . . 33 ( ( 5 ): ): 1641 - - 1654 . . DOI: 10.2514/1.46815 http://doi.org/10.2514/1.46815 . .

WB Powell . . Approximate Dynamic Programming: Solving the Curses of Dimensionality , , : : New Jersey, USA John Wiley & Sons , , 2007 . . .

SJ Russell , , , P Norvig . . Artificial Intelligence: a Modern Approach (3 $$.{\rm rd}$$ Ed.). Prentice Hall . . New Jersey, USA. , , 2010 . . .

F Sharifi , , , A Chamseddine , , , H Mahboubi , , , 等 . . A distributed deployment strategy for a network of cooperative autonomous vehicles . . IEEE Trans Contr Syst Technol , , 2016 . . 23 ( ( 2 ): ): 737 - - 745 . . DOI: 10.1109/TCST.2014.2341658 http://doi.org/10.1109/TCST.2014.2341658 . .

RS Sutton , , , AG Barto . . Reinforcement Learning: an Introduction , , : : Massachusetts, USA MIT Press , , 1998 . . .

C Thiery , , , B Scherrer . . Least-squares $$\lambda$$ policy iteration: bias-variance trade-off in control problems . . Proc 27 th Int Conf on Machine Learning , , 2010 . . p.1071 - - 1078 . . . .

B Wang , , , YM Zhang . . An adaptive fault-tolerant sliding mode control allocation scheme for multirotor helicopter subject to simultaneous actuator faults . . IEEE Trans Ind Electron , , 2018 . . 65 ( ( 5 ): ): 4227 - - 4236 . . DOI: 10.1109/TIE.2017.2772153 http://doi.org/10.1109/TIE.2017.2772153 . .

B Wang , , , X Yu , , , LX Mu , , , 等 . . Disturbance observer-based adaptive fault-tolerant control for a quadrotor helicopter subject to parametric uncertainties and external disturbances . . Mech Syst Signal Process , , 2019 . . 120 727 - - 743 . . DOI: 10.1016/j.ymssp.2018.11.001 http://doi.org/10.1016/j.ymssp.2018.11.001 . .

HZ Yu . . Convergence of least squares temporal difference methods under general conditions . . 27 th Int Conf on Machine Learning , , 2010 . . p.1207 - - 1214 . . . .

HZ Yu . . Least squares temporal difference methods: an analysis under general conditions . . SIAM J Contr Optim , , 2012 . . 50 ( ( 6 ): ): 3310 - - 3343 . . DOI: 10.1137/100807879 http://doi.org/10.1137/100807879 . .

C Yuan , , , YM Zhang , , , ZX Liu . . A survey on technologies for automatic forest fire monitoring, detection, and fighting using unmanned aerial vehicles and remote sensing techniques . . Can J Forest Res , , 2015 . . 45 ( ( 7 ): ): 783 - - 792 . . DOI: 10.1139/cjfr-2014-0347 http://doi.org/10.1139/cjfr-2014-0347 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Safe navigation of quadrotors with jerk limited trajectory

An improved sliding mode control based on fuzzy logic for quadrotor unmanned aerial vehicles under unmatched uncertainty

Coverage performance of the multilayer UAV-terrestrial HetNet with CoMP transmission scheme

Decentralized fault-tolerant cooperative control of multiple UAVs with prescribed attitude synchronization tracking performance under directed communication topology

Theoretical foundation of a decision network for urban development

Related Author

Shu-peng LAI

Meng-lu LAN

Ya-xuan LI

Ben M. CHEN

Lei YAO

Xiaodong ZHOU

Jiawei CHI

Weiqing SUN

Related Institution

Department of Electrical and Computer Engineering, National University of Singapore

Graduate School for Integrative Science & Engineering, National University of Singapore

School of Control Science and Engineering, Shandong University

Department of Mechanical and Automation Engineering, Chinese University of Hong Kong

School of Mechanical Engineering, University of Shanghai for Science and Technology

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰