A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks

Yi-ning CHEN; Ni-qi LYU; Guang-hua SONG; Bo-wei YANG; Xiao-hong JIANG

doi:10.1631/FITEE.1900401

Your Location：

Home >

Browse articles >

A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks

Regular Papers | Updated：2022-05-19

- A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks
- 基于GPSR和Q网络的流量感知无人机ad-hoc网络路由协议
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 9, Pages: 1308-1320(2020)
- Affiliations：
  
  School of Aeronautics and Astronautics, Zhejiang University, Hangzhou 310027, China
  College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
- Author bio：
  
  [ "Yi-ning CHEN, E-mail, ch19930611@zju.edu.cn" ]
  [ "Ni-qi LYU, E-mail, lvniqi@gmail.com" ]
  Guang-hua SONG, E-mail: ghsong@zju.edu.c
  [ "Ni-qi LYU, E-mail, boweiy@zju.edu.cn" ]
  [ "Xiao-hong JIANG, E-mail, jiangxh@zju.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1900401
  CLC： TP183;TN919.72
- Received：07 August 2019，
  
  Revised：2020-;8-;5，
  
  Published：2020-09
- Accepted：
Scan QR Code
Yi-ning CHEN, Ni-qi LYU, Guang-hua SONG, et al. A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1308-1320.
DOI：

Yi-ning CHEN, Ni-qi LYU, Guang-hua SONG, et al. A traffic-aware Q-network enhanced routing protocol based on GPSR for unmanned aerial vehicle ad-hoc networks[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1308-1320. DOI： 10.1631/FITEE.1900401.

摘要

在流量密集的无人机ad-hoc网络中，流量拥塞会增加网络时延和丢包，大大限制网络性能。因此，需要一个流量平衡策略控制流量。本文提出TQNGPSR，一个基于GPSR和Q网络的流量感知无人机ad-hoc网络路由协议。该协议利用邻居节点的拥塞信息实现流量平衡，并用一种强化学习算法-Q网络算法-评价当前节点每条无线链接的质量。基于对这些链接的评估，该协议可在多个选择中做出合理决定，降低网络时延和丢包率。在仿真环境中测试TQNGPSR、AODV、OLSR、GPSR和QNGPSR。结果表明，相比于GPSR和QNGPSR，TQNGPSR有更高包到达率和更低端到端时延。在高节点密度场景中，TQNGPSR在包到达率、端到端时延和吞吐量上优于AODV和OLSR。

Abstract

In dense traffic unmanned aerial vehicle (UAV) ad-hoc networks

traffic congestion can cause increased delay and packet loss

{which limit} the performance of the networks; therefore

a traffic balancing strategy is required to control the traffic. In this study

we propose TQNGPSR

a traffic-aware Q-network enhanced geographic routing protocol based on greedy perimeter stateless routing (GPSR)

for UAV ad-hoc networks. The protocol enforces a traffic balancing strategy using the congestion information of neighbors

and evaluates the quality of a wireless link by the Q-network algorithm

which is a reinforcement learning algorithm. Based on the evaluation of each wireless link

the protocol makes routing decisions in multiple available choices to reduce delay and decrease packet loss. We simulate the performance of TQNGPSR and compare it with AODV

OLSR

GPSR

and QNGPSR. Simulation results show that TQNGPSR obtains higher packet delivery ratios and lower end-to-end delays than GPSR and QNGPSR. In high node density scenarios

it also outperforms AODV and OLSR in terms of the packet delivery ratio

end-to-end delay

and throughput.

关键词

Keywords

references

M Abadi , , , P Barham , , , JM Chen , , , 等 . . TensorFlow: a system for large-scale machine learning . . Proc 12 th USENIX Conf on Operating Systems Design and Implementation , , 2016 . . p.265 - - 283 . . . .

S Basagni , , , I Chlamtac , , , VR Syrotiuk , , , 等 . . A distance routing effect algorithm for mobility (DREAM) . . Proc 4 th Annual ACM/IEEE Int Conf on Mobile Computing and Networking , , 1998 . . p.76 - - 84 . . DOI: 10.1145/288235.288254 http://doi.org/10.1145/288235.288254 . .

I Bekmezci , , , OK Sahingoz , , , Ş Temel . . Flying ad-hoc networks (FANETs): a survey . . Ad Hoc Netw , , 2013 . . 11 ( ( 3 ): ): 1254 - - 1270 . . DOI: 10.1016/j.adhoc.2012.12.004 http://doi.org/10.1016/j.adhoc.2012.12.004 . .

G Bolch , , , S Greiner , , , H de Meer , , , 等 . . Queueing Networks and Markov Chains: Modeling and Performance Evaluation with Computer Science Applications (2 nd Ed . . John Wiley & Sons, New York, USA , , 2006 . . .

JA Boyan , , , ML Littman . . Packet routing in dynamically changing networks: a reinforcement learning approach . . Proc 7 th Int Conf on Neural Information Processing Systems , , 1994 . . p.671 - - 678 . . . .

N Coutinho , , , R Matos , , , C Marques , , , 等 . . Dynamic dual-reinforcement-learning routing strategies for quality of experience-aware wireless mesh networking . . Comput Netw , , 2015 . . 88 269 - - 285 . . DOI: 10.1016/j.comnet.2015.06.016 http://doi.org/10.1016/j.comnet.2015.06.016 . .

F Farahnakian , , , M Ebrahimi , , , M Daneshtalab , , , 等 . . Q-learning based congestion-aware routing algorithm for on-chip network . . Proc 2 nd Int Conf on Networked Embedded Systems for Enterprise Applications , , 2011 . . p.1 - - 7 . . DOI: 10.1109/NESEA.2011.6144949 http://doi.org/10.1109/NESEA.2011.6144949 . .

WS Jung , , , J Yim , , , YB Ko . . QGeo: Q-learning-based geographic ad hoc routing protocol for unmanned robotic networks . . IEEE Commun Lett , , 2017 . . 21 ( ( 10 ): ): 2258 - - 2261 . . DOI: 10.1109/LCOMM.2017.2656879 http://doi.org/10.1109/LCOMM.2017.2656879 . .

B Karp , , , HT Kung . . GPSR: greedy perimeter stateless routing for wireless networks . . Proc 6 th Annual Int Conf on Mobile Computing and Networking , , 2000 . . p.243 - - 254 . . DOI: 10.1145/345910.345953 http://doi.org/10.1145/345910.345953 . .

T Kenta , , , M Takeshi , , , K Shinya , , , 等 . . Experimental evaluation of an on-demand multipath routing protocol for video transmission in mobile ad hoc networks . . J Zhejiang Univ-Sci A , , 2006 . . 7 ( ( S1 ): ): 145 - - 150 . . DOI: 10.1631/jzus.2006.AS0145 http://doi.org/10.1631/jzus.2006.AS0145 . .

YB Ko , , , NH Vaidya . . Location-aided routing (LAR) in mobile ad hoc networks . . Wirel Netw , , 2000 . . 6 ( ( 4 ): ): 307 - - 321 . . DOI: 10.1023/A:1019106118419 http://doi.org/10.1023/A:1019106118419 . .

RL Li , , , F Li , , , X Li , , , 等 . . QGrid: Q-learning based routing protocol for vehicular ad hoc networks . . Proc 33 rd Int Performance Computing and Communications Conf , , 2014 . . p.1 - - 8 . . DOI: 10.1109/PCCC.2014.7017079 http://doi.org/10.1109/PCCC.2014.7017079 . .

SC Lin , , , P Wang , , , M Luo . . Control traffic balancing in software defined networks . . Comput Netw , , 2016 . . 106 260 - - 271 . . DOI: 10.1016/j.comnet.2015.08.004 http://doi.org/10.1016/j.comnet.2015.08.004 . .

N Lyu , , , GH Song , , , BW Yang , , , 等 . . QNGPSR: a Q-network enhanced geographic ad-hoc routing protocol based on GPSR . . Proc 88 th Vehicular Technology Conf , , 2019 . . p.1 - - 6 . . DOI: 10.1109/VTCFall.2018.8690651 http://doi.org/10.1109/VTCFall.2018.8690651 . .

X Ma , , , Y Xu , , , GQ Sun , , , 等 . . State-chain sequential feedback reinforcement learning for path planning of autonomous mobile robots . . J Zhejiang Univ-Sci C (Comput & Electron) , , 2013 . . 14 ( ( 3 ): ): 167 - - 178 . . DOI: 10.1631/jzus.C1200226 http://doi.org/10.1631/jzus.C1200226 . .

V Mnih , , , K Kavukcuoglu , , , D Silver , , , 等 . . Human-level control through deep reinforcement learning . . Nature , , 2015 . . 518 ( ( 7540 ): ): 529 - - 533 . . DOI: 10.1038/nature14236 http://doi.org/10.1038/nature14236 . .

J Peng , , , RJ Williams . . Incremental multi-step Q-learning . . Mach Learn , , 1996 . . 22 ( ( 1-3 ): ): 283 - - 290 . . DOI: 10.1007/BF00114731 http://doi.org/10.1007/BF00114731 . .

RH Shi , , , YY Deng . . An improved scheme for reducing the latency of AODV in ad hoc networks . . Proc 9 th Int Conf for Young Computer Scientists , , 2008 . . p.594 - - 598 . . DOI: 10.1109/ICYCS.2008.325 http://doi.org/10.1109/ICYCS.2008.325 . .

RS Sutton , , , AG Barto . . Reinforcement Learning: an Introduction . . MIT Press, Cambridge , , 1998 . . p.1 . .

CJCH Watkins , , , P Dayan . . Q-learning . . Mach Learn , , 1992 . . 8 ( ( 3-4 ): ): 279 - - 292 . . DOI: 10.1007/BF00992698 http://doi.org/10.1007/BF00992698 . .

C Wu , , , S Ohzahata , , , T Kato . . Routing in VANETs: a fuzzy constraint Q-learning approach . . Proc Global Communications Conf , , 2012 . . p.195 - - 200 . . DOI: 10.1109/GLOCOM.2012.6503112 http://doi.org/10.1109/GLOCOM.2012.6503112 . .

C Wu , , , S Ohzahata , , , T Kato . . Flexible, portable, and practicable solution for routing in VANETs: a fuzzy constraint Q-learning approach . . IEEE Trans Veh Technol , , 2013 . . 62 ( ( 9 ): ): 4251 - - 4263 . . DOI: 10.1109/TVT.2013.2273945 http://doi.org/10.1109/TVT.2013.2273945 . .

DH Xu , , , M Chiang , , , J Rexford . . Link-state routing with hop-by-hop forwarding can achieve optimal traffic engineering . . IEEE/ACM Trans Netw , , 2011 . . 19 ( ( 6 ): ): 1717 - - 1730 . . DOI: 10.1109/TNET.2011.2134866 http://doi.org/10.1109/TNET.2011.2134866 . .

HW Zhan , , , Y Zhou . . Comparison and analysis AODV and OLSR routing protocols in ad hoc network . . Proc 4 th Int Conf on Wireless Communications, Networking and Mobile Computing , , 2008 . . p.1 - - 4 . . DOI: 10.1109/WiCom.2008.578 http://doi.org/10.1109/WiCom.2008.578 . .

JJ Zhang , , , K Xi , , , M Luo , , , 等 . . Load balancing for multiple traffic matrices using SDN hybrid routing . . Proc 15 th Int Conf on High Performance Switching and Routing , , 2014 . . p.44 - - 49 . . http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=CC0214411200 http://www.wanfangdata.com.cn/details/detail.do?_type=perio&id=CC0214411200 , , . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Jiu fusion artificial intelligence (JFA): a two-stage reinforcement learning model with hierarchical neural networks and human knowledge for Tibetan Jiu chess

Efficient learning of robust multigait quadruped locomotion for minimizing the cost of transport

Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Multi-agent reinforcement learning behavioral control for nonlinear second-order systems

Related Author

Xiali LI

Xiaoyu FAN

Junzhi YU

Xianmu CAIRANG

Ping LAN

Zhicheng DONG

Zhicheng WANG

Xin ZHAO

Related Institution

School of Information Science and Technology, Xizang University

State Key Laboratory of General Artificial Intelligence, Peking University

State Key Laboratory for Turbulence and Complex Systems, Peking University

School of Information Engineering, Minzu University of China

Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰