Cooperative channel assignment for VANETs based on multiagent reinforcement learning

Yun-peng WANG; Kun-xian ZHENG; Da-xin TIAN; Xu-ting DUAN; Jian-shan ZHOU

doi:10.1631/FITEE.1900308

Your Location：

Home >

Browse articles >

Cooperative channel assignment for VANETs based on multiagent reinforcement learning

Regular Papers | Updated：2022-05-19

- Cooperative channel assignment for VANETs based on multiagent reinforcement learning
- 基于多智能体强化学习的车载自组织网络协作信道分配
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 7, Pages: 1047-1058(2020)
- Affiliations：
  
  Beijing Advanced Innovation Center for Big Data and Brain Computing, School of Transportation Science and Engineering, Beihang University, Beijing 100191, China
- Author bio：
  
  [ "Yun-peng WANG, E-mail: ypwang@buaa.edu.cn" ]
  Kun-xian ZHENG, E-mail: zhengkunxian@buaa.edu.cn
  [ "Da-xin TIAN, E-mail: dtian@buaa.edu.cn" ]
  [ "Xu-ting DUAN, E-mail: duanxuting@buaa.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1900308
  CLC： U495
- Received：21 June 2019，
  
  Revised：2020-;6-11，
  
  Published：2020-07
- Accepted：
Scan QR Code
Yun-peng WANG, Kun-xian ZHENG, Da-xin TIAN, et al. Cooperative channel assignment for VANETs based on multiagent reinforcement learning[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(7): 1047-1058.
DOI：

Yun-peng WANG, Kun-xian ZHENG, Da-xin TIAN, et al. Cooperative channel assignment for VANETs based on multiagent reinforcement learning[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(7): 1047-1058. DOI： 10.1631/FITEE.1900308.

摘要

动态信道分配（DCA）在扩展车载自组织网络容量和缓解其拥塞方面起着关键作用。然而，在车-车直连通信场景下，信道分配面临大规模节点相互影响、缺乏集中式协调、全局网络状态信息未知以及其他挑战。为解决该问题，提出一种基于多智能体强化学习（RL）的协作动态信道分配（RL-CDCA）机制。具体而言，每个车辆节点都可借助2个互相协作的RL模型，从实时信道状态信息中成功学习信道选择和信道接入自适应退避的正确策略。此外，将神经网络构造为非线性Q函数逼近器，有助于将感测到的连续输入值映射到混合策略输出。多智能体RL-CDCA驱动节点共享本地奖励并合并区域内其他节点各自的奖励，以便它们能够以分布式协作方式优化各自策略。仿真结果表明，与4种现有机制相比，所提多智能体RL-CDCA算法即便在路网车辆高度密集的情况下仍能将单跳数据包传输延迟减少不小于73.73％，将平均数据包递送成功率提高不小于12.66％，并更好地保证网络资源分配公平性。

Abstract

Dynamic channel assignment (DCA) plays a key role in extending vehicular ad-hoc network capacity and mitigating congestion. However

channel assignment under vehicular direct communication scenarios faces mutual influence of large-scale nodes

the lack of centralized coordination

unknown global state information

and other challenges. To solve this problem

a multiagent reinforcement learning (RL) based cooperative DCA (RL-CDCA) mechanism is proposed. Specifically

each vehicular node can successfully learn the proper strategies of channel selection and backoff adaptation from the real-time channel state information (CSI) using two cooperative RL models. In addition

neural networks are constructed as nonlinear Q-function approximators

which facilitates the mapping of the continuously sensed input to the mixed policy output. Nodes are driven to locally share and incorporate their individual rewards such that they can optimize their policies in a distributed collaborative manner. Simulation results show that the proposed multiagent RL-CDCA can better reduce the one-hop packet delay by no less than 73.73%

improve the packet delivery ratio by no less than 12.66% on average in a highly dense situation

and improve the fairness of the global network resource allocation.

关键词

Keywords

references

SAM Ahmed , , , SHS Ariffin , , , N Fisal . . Overview of wireless access in vehicular environment (wave) protocols and standards . . Ind J Sci Technol , , 2013 . . 7 ( ( 6 ): ): 4994 - - 5001 . . DOI: 10.17485/ijst/2013/v6i7/34355 http://doi.org/10.17485/ijst/2013/v6i7/34355 . .

T Ahmed , , , Y Le Moullec . . A QoS optimization approach in cognitive body area networks for healthcare applications . . Sensors , , 2017 . . 17 ( ( 4 ): ): 780 DOI: 10.3390/s17040780 http://doi.org/10.3390/s17040780 . .

T Ahmed , , , F Ahmed , , , Y Le Moullec . . Optimization of channel allocation in wireless body area networks by means of reinforcement learning . . IEEE Asia Pacific Conf on Wireless and Mobile , , 2017 . . p.120 - - 123 . . DOI: 10.1109/APWiMob.2016.7811445 http://doi.org/10.1109/APWiMob.2016.7811445 . .

AA Almohammedi , , , NK Noordin , , , A Sali , , , 等 . . An adaptive multi-channel assignment and coordination scheme for IEEE 802 . . 11p/1609.4 in vehicular ad-hoc networks. IEEE Access , , 2017 . . 6 2781 - - 2802 . . DOI: 10.1109/ACCESS.2017.2785309 http://doi.org/10.1109/ACCESS.2017.2785309 . .

K Arulkumaran , , , MP Deisenroth , , , M Brundage , , , 等 . . A brief survey of deep reinforcement learning . . IEEE Signal Process Mag , , 2017 . . 34 ( ( 6 ): ): 26 - - 38 . . DOI: 10.1109/MSP.2017.2743240 http://doi.org/10.1109/MSP.2017.2743240 . .

R Atallah , , , C Assi , , , M Khabbaz . . Deep reinforcement learning-based scheduling for roadside communication networks . . $15^{\rm {th}}$ Int Symp on Modeling and Optimization in Mobile , , 2017 . . p.1 - - 8 . . DOI: 10.23919/WIOPT.2017.7959912 http://doi.org/10.23919/WIOPT.2017.7959912 . .

GK Audhya , , , K Sinha , , , SC Ghosh , , , 等 . . A survey on the channel assignment problem in wireless networks . . Wirel Commun Mob Comput , , 2011 . . 11 ( ( 5 ): ): 583 - - 609 . . DOI: 10.1002/wcm.898 http://doi.org/10.1002/wcm.898 . .

AG Barto , , , RS Sutton . . Reinforcement Learning: an Introduction , , : : Cambridge, MA, USA MIT Press , , 1998 . . .

J Cheeneebash , , , JA Lozano , , , HCS Rughooputh . . A survey on the algorithms used to solve the channel assignment problem . . Rec Pat Telecommun , , 2012 . . 1 ( ( 1 ): ): 54 - - 71 . . DOI: 10.2174/2211740711201010054 http://doi.org/10.2174/2211740711201010054 . .

Y He , , , N Zhao , , , HX Yin . . Integrated networking, caching, and computing for connected vehicles: a deep reinforcement learning approach . . IEEE Trans Veh Technol , , 2017 . . 67 ( ( 1 ): ): 44 - - 55 . . DOI: 10.1109/TVT.2017.2760281 http://doi.org/10.1109/TVT.2017.2760281 . .

RK Jain , , , DMW Chiu , , , WR Hawe . . A Quantitative Measure of Fairness and Discrimination for Resource Allocation in Shared Computer Systems . . CoRR. cs. NI/9809099, DEC, Hudson, Canada , , 1998 . . .

LP Kaelbling , , , ML Littman , , , AW Moore . . Reinforcement learning: a survey . . J Artif Intell Res , , 1996 . . 4 ( ( 1 ): ): 237 - - 285 . . DOI: 10.1613/jair.301 http://doi.org/10.1613/jair.301 . .

L Li , , , YS Lv , , , FY Wang . . Traffic signal timing via deep reinforcement learning . . IEEE/CAA J Autom Sin , , 2016 . . 3 ( ( 3 ): ): 247 - - 254 . . DOI: 10.1109/JAS.2016.7508798 http://doi.org/10.1109/JAS.2016.7508798 . .

XH Li , , , BJ Hu , , , HB Chen , , , 等 . . An RSU-coordinated synchronous multi-channel MAC scheme for vehicular ad hoc networks . . IEEE Access , , 2015 . . 3 2794 - - 2802 . . DOI: 10.1109/ACCESS.2015.2509458 http://doi.org/10.1109/ACCESS.2015.2509458 . .

N Liu , , , Z Li , , , JL Xu , , , 等 . . A hierarchical framework of cloud resource allocation and power management using deep reinforcement learning . . IEEE $37^{\rm {th}}$ Int Conf on Distributed Computing Systems , , 2017 . . p.372 - - 382 . . DOI: 10.1109/ICDCS.2017.123 http://doi.org/10.1109/ICDCS.2017.123 . .

SJ Liu , , , X Hu , , , WD Wang . . Deep reinforcement learning based dynamic channel allocation algorithm in multibeam satellite systems . . IEEE Access , , 2018 . . 6 15733 - - 15742 . . DOI: 10.1109/ACCESS.2018.2809581 http://doi.org/10.1109/ACCESS.2018.2809581 . .

M Louta , , , P Sarigiannidis , , , S Misra , , , 等 . . RLAM: a dynamic and efficient reinforcement learning-based adaptive mapping scheme in mobile WiMAX networks . . Mob Inform Syst , , 2014 . . 10 ( ( 2 ): ): 173 - - 196 . . DOI: 10.1155/2014/213056 http://doi.org/10.1155/2014/213056 . .

CJ Maddison , , , A Huang , , , I Sutskever , , , 等 . . Move evaluation in go using deep convolutional neural networks . . 2014 . . https://arxiv.org/abs/1412.6564 https://arxiv.org/abs/1412.6564 , , . .

HZ Mao , , , M Alizadeh , , , I Menache , , , 等 . . Resource management with deep reinforcement learning . . Proc $15^{\rm {th}}$ ACM Workshop on Hot Topics in Networks , , 2016 . . p.50 - - 56 . . DOI: 10.1145/3005745.3005750 http://doi.org/10.1145/3005745.3005750 . .

V Mnih , , , K Kavukcuoglu , , , D Silver , , , 等 . . Playing Atari with deep reinforcement learning . . 2013 . . https://arxiv.org/abs/1312.5602 https://arxiv.org/abs/1312.5602 , , . .

V Mnih , , , K Kavukcuoglu , , , D Silver , , , 等 . . Human-level control through deep reinforcement learning . . Nature , , 2015 . . 518 ( ( 7540 ): ): 529 - - 533 . . DOI: 10.1038/nature14236 http://doi.org/10.1038/nature14236 . .

JH Nie , , , S Haykin . . A dynamic channel assignment policy through Q-learning . . IEEE Trans Neur Netw , , 1999 . . 10 ( ( 6 ): ): 1443 - - 1455 . . DOI: 10.1109/72.809089 http://doi.org/10.1109/72.809089 . .

M Ouyous , , , O Zytoune , , , D Aboutajdine . . Multi-channel coordination based MAC protocols in vehicular ad hoc networks (VANETs): a survey . . In: El-Azouzi R, Menasche D, Sabir E, et al. (Eds.), Advances in Ubiquitous Networking 2. Springer, Singapore , , 2017 . . DOI: 10.1007/978-981-10-1627-1_7 http://doi.org/10.1007/978-981-10-1627-1_7 . .

CR Qiu , , , Y Hu , , , Y Chen , , , 等 . . Deep deterministic policy gradient (DDPG)-based energy harvesting wireless communications . . IEEE Int Things J , , 2019 . . 6 ( ( 5 ): ): 8577 - - 8588 . . DOI: 10.1109/JIOT.2019.2921159 http://doi.org/10.1109/JIOT.2019.2921159 . .

MWM Seah , , , CK Tham , , , V Srinivasan , , , 等 . . Achieving coverage through distributed reinforcement learning in wireless sensor networks . . $3^{\rm {rd}}$ Int Conf on Intelligent Sensors, Sensor Networks and Information , , 2007 . . p.425 - - 430 . . DOI: 10.1109/ISSNIP.2007.4496881 http://doi.org/10.1109/ISSNIP.2007.4496881 . .

D Silver , , , J Schrittwieser , , , K Simonyan , , , 等 . . Mastering the game of go without human knowledge . . Nature , , 2017 . . 550 ( ( 7676 ): ): 354 - - 350 . . DOI: 10.1038/nature24270 http://doi.org/10.1038/nature24270 . .

Q Wang , , , S Leng , , , HR Fu , , , 等 . . An IEEE 802 . . 11p-based multichannel MAC scheme with channel coordination for vehicular ad hoc networks. IEEE Trans Intell Trans Syst , , 2012 . . 13 ( ( 2 ): ): 449 - - 458 . . DOI: 10.1109/tits.2011.2171951 http://doi.org/10.1109/tits.2011.2171951 . .

W Wang , , , A Kwasinski , , , D Niyato , , , 等 . . A survey on applications of model-free strategy learning in cognitive wireless networks . . IEEE Commun Surv Tutor , , 2017 . . 18 ( ( 3 ): ): 1717 - - 1757 . . DOI: 10.1109/COMST.2016.2539923 http://doi.org/10.1109/COMST.2016.2539923 . .

ZY Xu , , , YZ Wang , , , J Tang , , , 等 . . A deep reinforcement learning based framework for power-efficient resource allocation in cloud RANs . . IEEE Int Conf on Communications , , 2017 . . p.1 - - 6 . . DOI: 10.1109/ICC.2017.7997286 http://doi.org/10.1109/ICC.2017.7997286 . .

KLA Yau , , , P Komisarczuk , , , DT Paul . . Enhancing network performance in distributed cognitive radio networks using single-agent and multi-agent reinforcement learning . . IEEE Local Computer Network Conf , , 2010 . . p.152 - - 159 . . DOI: 10.1109/LCN.2010.5735689 http://doi.org/10.1109/LCN.2010.5735689 . .

H Ye , , , GY Li , , , BHF Juang . . Deep reinforcement learning based resource allocation for V2V communications . . IEEE Int Conf on Communications , , 2018 . . p.1 - - 6 . . DOI: 10.1109/ICC.2018.8422586 http://doi.org/10.1109/ICC.2018.8422586 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Jiu fusion artificial intelligence (JFA): a two-stage reinforcement learning model with hierarchical neural networks and human knowledge for Tibetan Jiu chess

Efficient learning of robust multigait quadruped locomotion for minimizing the cost of transport

Reinforcement learning based privacy-preserving consensus tracking control of nonstrict-feedback discrete-time multi-agent systems

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Multi-agent reinforcement learning behavioral control for nonlinear second-order systems

Related Author

Xiali LI

Xiaoyu FAN

Junzhi YU

Xianmu CAIRANG

Ping LAN

Zhicheng DONG

Zhicheng WANG

Xin ZHAO

Related Institution

School of Information Science and Technology, Xizang University

State Key Laboratory of General Artificial Intelligence, Peking University

State Key Laboratory for Turbulence and Complex Systems, Peking University

School of Information Engineering, Minzu University of China

Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰