Image-based traffic signal control via world models

Xingyuan DAI; Chen ZHAO; Xiao WANG; Yisheng LV; Yilun LIN; Fei-Yue WANG

doi:10.1631/FITEE.2200323

Your Location：

Home >

Browse articles >

Image-based traffic signal control via world models

Regular Papers | Updated：2022-12-16

- Image-based traffic signal control via world models
- 基于世界模型与图像表示的交通信号控制
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 12, Pages: 1795-1813(2022)
- Affiliations：
  
  1.The State Key Laboratory for Management and Control of Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing 100190, China
  2.School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing 100049, China
  3.School of Artificial Intelligence, Anhui University, Hefei 230039, China
  4.Shanghai AI Laboratory, Shanghai 200232, China
- Author bio：
  
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2200323
  CLC： U491; TP181
- Published：0 December 2022，
  
  Received：28 July 2022，
  
  Accepted：2022-10-06
- Accepted：
Scan QR Code
XINGYUAN DAI, CHEN ZHAO, XIAO WANG, et al. Image-based traffic signal control via world models. [J]. Frontiers of information technology & electronic engineering, 2022, 23(12): 1795-1813.
DOI：

XINGYUAN DAI, CHEN ZHAO, XIAO WANG, et al. Image-based traffic signal control via world models. [J]. Frontiers of information technology & electronic engineering, 2022, 23(12): 1795-1813. DOI： 10.1631/FITEE.2200323.

摘要

交通信号控制正从被动控制过渡到主动控制，以引导当前交通流按预期状态运行。一个有效的预测模型对主动交通信号控制至关重要；其中预测什么交通状态，如何高精度预测，以及如何利用预测优化控制策略是主动交通信号控制研究的关键问题。本文使用车辆位置图像描述路口交通状态，同时受基于模型的强化学习方法DreamerV2的启发，引入基于学习的交通世界模型。该世界模型以图像序列描述交通动态，并作为交通环境的抽象替代以生成多步预测样本用于控制策略优化。在执行阶段，优化后的交通信号控制器根据交通状态的抽象表示直接实时输出控制指令，同时世界模型能够预测不同控制行为对未来交通状态的影响。实验结果表明，基于交通世界模型优化的控制策略的性能优于一般基准，并且世界模型实现了基于图像的高精度预测；这些结果显示了世界模型在未来交通信号控制中的应用前景。

Abstract

Traffic signal control is shifting from passive control to proactive control

which enables the controller to direct current traffic flow to reach its expected destinations. To this end

an effective prediction model is needed for signal controllers. What to predict

how to predict

and how to leverage the prediction for control policy optimization are critical problems for proactive traffic signal control. In this paper

we use an image that contains vehicle positions to describe intersection traffic states. Then

inspired by a model-based reinforcement learning method

DreamerV2

we introduce a novel learning-based traffic world model. The traffic world model that describes traffic dynamics in image form is used as an abstract alternative to the traffic environment to generate multi-step planning data for control policy optimization. In the execution phase

the optimized traffic controller directly outputs actions in real time based on abstract representations of traffic states

and the world model can also predict the impact of different control behaviors on future traffic conditions. Experimental results indicate that the traffic world model enables the optimized real-time control policy to outperform common baselines

and the model achieves accurate image-based prediction

showing promising applications in futuristic traffic signal control.

关键词

交通信号控制交通预测交通世界模型强化学习

Keywords

Traffic signal controlTraffic predictionTraffic world modelReinforcement learning

references

Abdoos M, Bazzan ALC, 2021. Hierarchical traffic signal optimization using reinforcement learning and traffic prediction with long-short term memory. Expert Syst Appl, 171:114580. doi: 10.1016/j.eswa.2021.114580http://doi.org/10.1016/j.eswa.2021.114580

Bertsekas D, 2021. Multiagent reinforcement learning: rollout and policy iteration. IEEE/CAA J Autom Sin, 8(2):249-272. doi: 10.1109/JAS.2021.1003814http://doi.org/10.1109/JAS.2021.1003814

Dai XY, Fu R, Zhao EM, et al., 2019. DeepTrend 2.0: a light-weighted multi-scale traffic prediction model using detrending. Transp Res Part C Emerg Technol, 103:142-157. doi: 10.1016/j.trc.2019.03.022http://doi.org/10.1016/j.trc.2019.03.022

Guo QQ, Li L, Ban XG, 2019. Urban traffic signal control with connected and automated vehicles: a survey. Transp Res Part C Emerg Technol, 101:313-334. doi: 10.1016/j.trc.2019.01.026http://doi.org/10.1016/j.trc.2019.01.026

Hafner D, Lillicrap T, Fischer I, et al., 2019. Learning latent dynamics for planning from pixels. Proc 36th Int Conf on Machine Learning, p.2555-2565.

Hafner D, Lillicrap TP, Norouzi M, et al., 2022. Mastering Atari with discrete world models. https://arxiv.org/abs/2010.02193https://arxiv.org/abs/2010.02193

Hao ZZ, Boel R, Li ZW, 2018. Model based urban traffic control, part I: local model and local model predictive controllers. Transp Res Part C Emerg Technol, 97:61-81. doi: 10.1016/j.trc.2018.09.026http://doi.org/10.1016/j.trc.2018.09.026

Jin JC, Guo HF, Xu J, et al., 2021. An end-to-end recommendation system for urban traffic controls and management under a parallel learning framework. IEEE Trans Intell Transp Syst, 22(3):1616-1626. doi: 10.1109/TITS.2020.2973736http://doi.org/10.1109/TITS.2020.2973736

Kim D, Jeong O, 2019. Cooperative traffic signal control with traffic flow prediction in multi-intersection. Sensors, 20(1):137. doi: 10.3390/s20010137http://doi.org/10.3390/s20010137

Li L, Lv YS, Wang FY, 2016. Traffic signal timing via deep reinforcement learning. IEEE/CAA J Autom Sin, 3(3):247-254. doi: 10.1109/JAS.2016.7508798http://doi.org/10.1109/JAS.2016.7508798

Li L, Lin YL, Zheng NN, et al., 2017. Parallel learning: a perspective and a framework. IEEE/CAA J Autom Sin, 4(3):389-395. doi: 10.1109/JAS.2017.7510493http://doi.org/10.1109/JAS.2017.7510493

Li ZS, Xiong G, Tian YL, et al., 2022. A multi-stream feature fusion approach for traffic prediction. IEEE Trans Intell Transp Syst, 23(2):1456-1466. doi: 10.1109/TITS.2020.3026836http://doi.org/10.1109/TITS.2020.3026836

Liang XY, Du XS, Wang GL, et al., 2019. A deep reinforcement learning network for traffic light cycle control. IEEE Trans Veh Technol, 68(2):1243-1253. doi: 10.1109/TVT.2018.2890726http://doi.org/10.1109/TVT.2018.2890726

Liu CH, Zhu F, Liu Q, et al., 2021. Hierarchical reinforcement learning with automatic sub-goal identification. IEEE/CAA J Autom Sin, 8(10):1686-1696. doi: 10.1109/JAS.2021.1004141http://doi.org/10.1109/JAS.2021.1004141

Lopez PA, Behrisch M, Bieker-Walz L, et al., 2018. Microscopic traffic simulation using SUMO. Proc 21st IEEE Int Conf on Intelligent Transportation Systems, p.2575-2582. doi: 10.1109/ITSC.2018.8569938http://doi.org/10.1109/ITSC.2018.8569938

Lv YS, Duan YJ, Kang WW, et al., 2014. Traffic flow prediction with big data: a deep learning approach. IEEE Trans Intell Transp Syst, 16(2):865-873. doi: 10.1109/TITS.2014.2345663http://doi.org/10.1109/TITS.2014.2345663

Mao F, Li ZH, Li L, 2022. A comparison of deep reinforcement learning models for isolated traffic signal control. IEEE Intell Transp Syst Mag, early access. doi: 10.1109/MITS.2022.3144797http://doi.org/10.1109/MITS.2022.3144797

Mei ZY, Tan Z, Zhang W, et al., 2019. Simulation analysis of traffic signal control and transit signal priority strategies under arterial coordination conditions. Simulation, 95(1):51-64. doi: 10.1177/0037549718757651http://doi.org/10.1177/0037549718757651

Mnih V, Kavukcuoglu K, Silver D, et al., 2015. Human-level control through deep reinforcement learning. Nature, 518(7540):529-533. doi: 10.1038/nature14236http://doi.org/10.1038/nature14236

Newell GF, 1969. Properties of vehicle-actuated signals: I. one-way streets. Transp Sci, 3(1):30-52.

Nie J, Yan J, Yin HL, et al., 2021. A multimodality fusion deep neural network and safety test strategy for intelligent vehicles. IEEE Trans Intell Veh, 6(2):310-322. doi: 10.1109/TIV.2020.3027319http://doi.org/10.1109/TIV.2020.3027319

Seng D, Lv FS, Liang ZY, et al., 2021. Forecasting traffic flows in irregular regions with multi-graph convolutional network and gated recurrent unit. Front Inform Technol Electron Eng, 22(9):1179-1193. doi: 10.1631/FITEE.2000243http://doi.org/10.1631/FITEE.2000243

Sutton RS, Barto AG, 2018. Reinforcement Learning: an Introduction (2nd Ed.). The MIT Press, Cambridge, USA.

Varaiya P, 2013. Max pressure control of a network of signalized intersections. Transp Res Part C Emerg Technol, 36:177-195. doi: 10.1016/j.trc.2013.08.014http://doi.org/10.1016/j.trc.2013.08.014

Wang FY, 2010. Parallel control and management for intelligent transportation systems: concepts, architectures, and applications. IEEE Trans Intell Transp Syst, 11(3):630-638. doi: 10.1109/TITS.2010.2060218http://doi.org/10.1109/TITS.2010.2060218

Wang HN, Liu N, Zhang YY, et al., 2020. Deep reinforcement learning: a survey. Front Inform Technol Electron Eng, 21(12):1726-1744. doi: 10.1631/FITEE.1900533http://doi.org/10.1631/FITEE.1900533

Wang J, Li R, Wang J, et al., 2020. Artificial intelligence and wireless communications. Front Inform Technol Electron Eng, 21(10):1413-1425. doi: 10.1631/FITEE.1900527http://doi.org/10.1631/FITEE.1900527

Webster FV, 1958. Traffic Signal Settings. Technical Report No. 39, Road Research Laboratory, UK.

Wei H, Xu N, Zhang HC, et al., 2019a. CoLight: learning network-level cooperation for traffic signal control. Proc 28th ACM Int Conf on Information and Knowledge Management, p.1913-1922. doi: 10.1145/3357384.3357902http://doi.org/10.1145/3357384.3357902

Wei H, Chen CC, Zheng GJ, et al., 2019b. PressLight: learning max pressure control to coordinate traffic signals in arterial network. Proc 25th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining, p.1290-1298. doi: 10.1145/3292500.3330949http://doi.org/10.1145/3292500.3330949

Wiering M, 2000. Multi-agent reinforcement learning for traffic light control. Proc 17th Int Conf on Machine Learning, p.1151-1158.

Xiao Y, Codevilla F, Gurram A, et al., 2022. Multimodal end-to-end autonomous driving. IEEE Trans Intell Transp Syst, 23(1):537-547. doi: 10.1109/TITS.2020.3013234http://doi.org/10.1109/TITS.2020.3013234

Xiong G, Dong XS, Lu H, et al., 2020. Research progress of parallel control and management. IEEE/CAA J Autom Sin, 7(2):355-367. doi: 10.1109/JAS.2019.1911792http://doi.org/10.1109/JAS.2019.1911792

Ye BL, Wu WM, Ruan KY, et al., 2019. A survey of model predictive control methods for traffic signal control. IEEE/CAA J Autom Sin, 6(3):623-640. doi: 10.1109/JAS.2019.1911471http://doi.org/10.1109/JAS.2019.1911471

Yu ZX, Liang SX, Wei L, et al., 2020. MaCAR: urban traffic light control via active multi-agent communication and action rectification. Proc 29th Int Joint Conf on Artificial Intelligence, p.2491-2497. doi: 10.24963/ijcai.2020/345http://doi.org/10.24963/ijcai.2020/345

Zhang HC, Kafouros M, Yu Y, 2020. PlanLight: learning to optimize traffic signal control with planning and iterative policy improvement. IEEE Access, 8:219244-219255. doi: 10.1109/ACCESS.2020.3041441http://doi.org/10.1109/ACCESS.2020.3041441

Zhang KQ, Yang ZR, Basar T, 2021. Decentralized multi-agent reinforcement learning with networked agents: recent advances. Front Inform Technol Electron Eng, 22(6):802-814. doi: 10.1631/FITEE.1900661http://doi.org/10.1631/FITEE.1900661

Zhao YF, Gao H, Wang S, et al., 2017. A novel approach for traffic signal control: a recommendation perspective. IEEE Intell Transp Syst Mag, 9(3):127-135. doi: 10.1109/MITS.2017.2709779http://doi.org/10.1109/MITS.2017.2709779

Zhu FH, Lv YS, Chen YY, et al., 2020. Parallel transportation systems: toward IoT-enabled smart urban traffic control and management. IEEE Trans Intell Transp Syst, 21(10):4063-4071. doi: 10.1109/TITS.2019.2934991http://doi.org/10.1109/TITS.2019.2934991

Views

112

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Domain adaptation in reinforcement learning: a comprehensive and systematic study

Multi-agent reinforcement learning behavioral control for nonlinear second-order systems

An anti-collision algorithm for robotic search-and-rescue tasks in unknown dynamic environments

Embedding expert demonstrations into clustering buffer for effective deep reinforcement learning

A home energy management approach using decoupling value and policy in reinforcement learning

Related Author

Amirfarhad FARHADI

Mitra MIRZAREZAEE

Arash SHARIFI

Mohammad TESHNEHLAB

Zhenyi ZHANG

Jie HUANG

Congjie PAN

Yang CHEN

Related Institution

Department of Control Engineering, K.N. Toosi University of Technology

Department of Computer Engineering, Science and Research Branch, Islamic Azad University

G+ Industrial Internet Institute, Fuzhou University

College of Electrical Engineering and Automation, Fuzhou University

College of Computer, National University of Defense Technology

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰