A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars

Jian-ru XUE; Di WANG; Shao-yi DU; Di-xiao CUI; Yong HUANG; Nan-ning ZHENG

doi:10.1631/FITEE.1601873

Your Location：

Home >

Browse articles >

A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars

Special Issue on Artificial Intelligence 2.0 | Updated：2022-05-19

- A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars
- 无人车自主定位和障碍物感知的视觉主导多传感器融合方法
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 1, Pages: 122-138(2017)
- Affiliations：
  
  Lab of Visual Cognitive Computing and Intelligent Vehicle, Xi'an Jiaotong University, Xi'an 710049, China
- Author bio：
  
  E-mail:jrxue@xjtu.edu.cn
- Funds：
- DOI：10.1631/FITEE.1601873
  CLC： TP181
- Received：29 December 2016，
  
  Accepted：08 January 2017，
  
  Published：2017-01
- Accepted：
Scan QR Code
Jian-ru XUE, Di WANG, Shao-yi DU, et al. A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(1): 122-138.
DOI：

Jian-ru XUE, Di WANG, Shao-yi DU, et al. A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(1): 122-138. DOI： 10.1631/FITEE.1601873.

摘要

人类驾驶与自主驾驶在对交通环境的理解方式上有着明显差别。首先，人主要通过视觉来理解交通场景，而机器感知需要融合多种异构的传感信息才能保证行车安全。其次，一个熟练的驾驶员可以轻松适应各种动态交通环境，但现有的机器感知系统却会经常输出有噪声的感知结果，而自主驾驶要求感知结果近乎100%准确。本文提出了一种用于无人车交通环境感知的视觉主导的多传感器融合计算框架，通过几何和语义约束融合来自相机、激光雷达（LIDAR）及地理信息系统（GIS）的信息，为无人车提供高精度的自主定位和准确鲁棒的障碍物感知，并进一步讨论了已成功集成到上述框架内的鲁棒的视觉算法，主要包括从训练数据收集、传感器数据处理、低级特征提取到障碍物识别和环境地图创建等多个层次的视觉算法。所提出的框架里已用于自主研发的无人车，并在各种真实城区环境中进行了长达八年的实地测试，实验结果验证了视觉主导的多传感融合感知框架的鲁棒性和高效性。

Abstract

Most state-of-the-art robotic cars' perception systems are quite different from the way a human driver understands traffic environments. First

humans assimilate information from the traffic scene mainly through visual perception

while the machine perception of traffic environments needs to fuse information from several different kinds of sensors to meet safety-critical requirements. Second

a robotic car requires nearly 100% correct perception results for its autonomous driving

while an experienced human driver works well with dynamic traffic environments

in which machine perception could easily produce noisy perception results. In this paper

we propose a vision-centered multi-sensor fusing framework for a traffic environment perception approach to autonomous driving

which fuses camera

LIDAR

and GIS information consistently via both geometrical and semantic constraints for efficient self-localization and obstacle perception. We also discuss robust machine vision algorithms that have been successfully integrated with the framework and address multiple levels of machine vision techniques

from collecting training data

efficiently processing sensor data

and extracting low-level features

to higher-level object and environment mapping. The proposed framework has been tested extensively in actual urban scenes with our self-developed robotic cars for eight years. The empirical results validate its robustness and efficiency.

关键词

Keywords

references

M Aeberhard , , , S Rauch , , , M Bahram , , , 等 . . Experience, results and lessons learned from automated driving on Germany's highways . . IEEE Intell. Transp. Syst. Mag. , , 2015 . . 7 ( ( 1 ): ): 42 - - 57 . . DOI: 10.1109/MITS.2014.2360306 http://doi.org/10.1109/MITS.2014.2360306 . .

JL Blanco , , , JA Fern dez-Madrigal , , , J Gonz lez . . A new approach for large-scale localization and mapping: hybrid metric-topological SLAM . . 2007 . . Proc. IEEE Int. Conf. on Robotics and Automation . . 2061 - - 2067 . . DOI: 10.1109/ROBOT.2007.363625 http://doi.org/10.1109/ROBOT.2007.363625 . .

JL Blanco , , , JA Fern dez-Madrigal , , , J Gonz lez . . Toward a unified Bayesian approach to hybrid metrictopological SLAM . . IEEE Trans. Robot. , , 2008 . . 24 ( ( 2 ): ): 259 - - 270 . . DOI: 10.1109/TRO.2008.918049 http://doi.org/10.1109/TRO.2008.918049 . .

MA Brubaker , , , A Geiger , , , R Urtasun . . Map-based probabilistic visual self-localization . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2016 . . 38 ( ( 4 ): ): 652 - - 665 . . DOI: 10.1109/TPAMI.2015.2453975 http://doi.org/10.1109/TPAMI.2015.2453975 . .

M Buehler , , , K Iagnemma , , , S Singh . . The DARPA Urban Challenge: Autonomous Vehicles in City Traffic , , 2009 . . : Springer : DOI: 10.1007/978-3-642-03991-1 http://doi.org/10.1007/978-3-642-03991-1 . .

H Cho , , , YW Seo , , , B.V.K.V Kumar , , , 等 . . A multi-sensor fusion system for moving object detection and tracking in urban driving environments . . 2014 . . IEEE Int. Conf. on Robotics and Automation . . 1836 - - 1843 . . DOI: 10.1109/ICRA.2014.6907100 http://doi.org/10.1109/ICRA.2014.6907100 . .

D Cui , , , J Xue , , , S Du , , , 等 . . Real-time global localization of intelligent road vehicles in lane-level via lane marking detection and shape registration . . 2014 . . IEEE/RSJ Int. Conf. on Intelligent Robots and Systems . . 4958 - - 4964 . . DOI: 10.1109/IROS.2014.6943267 http://doi.org/10.1109/IROS.2014.6943267 . .

DX Cui , , , JR Xue , , , NN Zheng . . Real-time global localization of robotic cars in lane level via lane marking detection and shape registration . . IEEE Trans. Intell. Transp. Syst. , , 2016 . . 17 ( ( 4 ): ): 1039 - - 1050 . . DOI: 10.1109/TITS.2015.2492019 http://doi.org/10.1109/TITS.2015.2492019 . .

M Darms , , , P Rybski , , , C Urmson . . Classification and tracking of dynamic objects with multiple sensors for autonomous driving in urban environments . . 2008 . . IEEE Intelligent Vehicles Symp . . 1197 - - 1202 . . DOI: 10.1109/IVS.2008.4621259 http://doi.org/10.1109/IVS.2008.4621259 . .

M Darms , , , P Rybski , , , C Baker , , , 等 . . Obstacle detection and tracking for the urban challenge . . IEEE Trans. Intell. Transp. Syst. , , 2009 . . 10 ( ( 3 ): ): 475 - - 485 . . DOI: 10.1109/TITS.2009.2018319 http://doi.org/10.1109/TITS.2009.2018319 . .

A J Davison , , , I D Reid , , , N D Molton , , , 等 . . MonoSLAM: real-time single camera SLAM . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2007 . . 29 ( ( 6 ): ): 1052 - - 1067 . . DOI: 10.1109/TPAMI.2007.1049 http://doi.org/10.1109/TPAMI.2007.1049 . .

M.W.M.G Dissanayake , , , P Newman , , , S Clark , , , 等 . . A solution to the simultaneous localization and map building (SLAM) problem . . IEEE Trans. Robot. Autom. , , 2001 . . 17 ( ( 3 ): ): 229 - - 241 . . DOI: 10.1109/70.938381 http://doi.org/10.1109/70.938381 . .

P Dollr , , , R Appel , , , S Belongie , , , 等 . . Fast feature pyramids for object detection . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2014 . . 36 ( ( 8 ): ): 1532 - - 1545 . . DOI: 10.1109/TPAMI.2014.2300479 http://doi.org/10.1109/TPAMI.2014.2300479 . .

B Douillard , , , D Fox , , , F Ramos . . Laser and vision based outdoor object mapping . . 2009 . . Robotics: Science and Systems Ⅳ . . 9 - - 16 . . . .

SY Du , , , NN Zheng , , , L Xiong , , , 等 . . Scaling iterative closest point algorithm for registration of m-D point sets . . J. Vis. Commun. Image Represent. , , 2010 . . 21 ( ( 5-6 ): ): 442 - - 452 . . DOI: 10.1016/j.jvcir.2010.02.005 http://doi.org/10.1016/j.jvcir.2010.02.005 . .

H Durrant-Whyte , , , T Bailey . . Simultaneous localization and mapping: part I . . IEEE Robot. Autom. Mag. , , 2006 . . 13 ( ( 2 ): ): 99 - - 110 . . DOI: 10.1109/MRA.2006.1638022 http://doi.org/10.1109/MRA.2006.1638022 . .

A Ess , , , K Schindler , , , B Leibe , , , 等 . . Object detection and tracking for autonomous navigation in dynamic environments . . Int. J. Robot. Res. , , 2010 . . 29 ( ( 14 ): ): 1707 - - 1725 . . DOI: 10.1177/0278364910365417 http://doi.org/10.1177/0278364910365417 . .

J Fuentes-Pacheco , , , J Ruiz-Ascencio , , , JM Rend n-Mancha . . Visual simultaneous localization and mapping: a survey . . Artif. Intell. Rev. , , 2015 . . 43 ( ( 1 ): ): 55 - - 81 . . DOI: 10.1007/s10462-012-9365-8 http://doi.org/10.1007/s10462-012-9365-8 . .

G Grisetti , , , R Kummerle , , , C Stachniss , , , 等 . . A tutorial on graph-based SLAM . . IEEE Intell. Transp. Syst. Mag. , , 2010 . . 2 ( ( 4 ): ): 31 - - 43 . . DOI: 10.1109/MITS.2010.939925 http://doi.org/10.1109/MITS.2010.939925 . .

RI Hartley , , , A Zisserman . . Multiple View Geometry in Computer Vision , , 2004 . . : Cambridge University Press : . .

D Held , , , J Levinson , , , S Thrun , , , 等 . . Robust realtime tracking combining 3D shape, color, and motion . . Int. J. Robot. Res. , , 2016 . . 35 ( ( 1-3 ): ): 30 - - 49 . . DOI: 10.1177/0278364915593399 http://doi.org/10.1177/0278364915593399 . .

AB Hillel , , , R Lerner , , , D Levi , , , 等 . . Recent progress in road and lane detection: a survey . . Mach. Vis. Appl. , , 2014 . . 25 ( ( 3 ): ): 727 - - 745 . . DOI: 10.1007/s00138-011-0404-2 http://doi.org/10.1007/s00138-011-0404-2 . .

D Hoiem , , , J Hays , , , JX Xiao , , , 等 . . Guest editorial: scene understanding . . Int. J. Comput. Vis. , , 2015 . . 112 ( ( 2 ): ): 131 - - 132 . . DOI: 10.1007/s11263-015-0807-z http://doi.org/10.1007/s11263-015-0807-z . .

K Konolige , , , E Marder-Eppstein , , , B Marthi . . Navigation in hybrid metric-topological maps . . 2011 . . IEEE Int. Conf. on Robotics and Automation . . 3041 - - 3047 . . DOI: 10.1109/ICRA.2011.5980074 http://doi.org/10.1109/ICRA.2011.5980074 . .

Q Li , , , NN Zheng , , , H Cheng . . Springrobot: a prototype autonomous vehicle and its algorithms for lane detection . . IEEE Trans. Intell. Transp. Syst. , , 2004 . . 5 ( ( 4 ): ): 300 - - 308 . . DOI: 10.1109/TITS.2004.838220 http://doi.org/10.1109/TITS.2004.838220 . .

C Mertz , , , LE Navarro-Serment , , , RA MacLachlan , , , 等 . . Moving object detection with laser scanners . . J. Field Robot. , , 2013 . . 30 ( ( 1 ): ): 17 - - 43 . . DOI: 10.1002/rob.21430 http://doi.org/10.1002/rob.21430 . .

M Montemerlo , , , S Thrun , , , D Koller , , , 等 . . FastSLAM: a factored solution to the simultaneous localization and mapping problem . . 2002 . . 8th National Conf. on Artificial Intelligence . . 593 - - 598 . . . .

YH Pan . . Heading toward artificial intelligence 2.0 . . Engineering , , 2016 . . 2 ( ( 4 ): ): 409 - - 413 . . DOI: 10.1016/J.ENG.2016.04.018 http://doi.org/10.1016/J.ENG.2016.04.018 . .

K Schueler , , , T Weiherer , , , E Bouzouraa , , , 等 . . 360 degree multi sensor fusion for static and dynamic obstacles . . 2012 . . IEEE Intelligent Vehicles Symp . . 692 - - 697 . . DOI: 10.1109/IVS.2012.6232253 http://doi.org/10.1109/IVS.2012.6232253 . .

S Thrun , , , JJ Leonard . . Simultaneous localization and mapping . . 2008 . . Int. Conf. on Artificial Intelligence . . 871 - - 889 . . DOI: 10.1007/978-3-540-30301-5_38 http://doi.org/10.1007/978-3-540-30301-5_38 . .

L Ulrich . . 2016's Top Ten Tech Cars . . 2016 . . Available from: http://spectrum.ieee.org/transportation/advanced-cars/2016s-top-ten-tech-cars http://spectrum.ieee.org/transportation/advanced-cars/2016s-top-ten-tech-cars . .

J Xue , , , NN Zheng , , , J Geng , , , 等 . . Tracking multiple visual targets via particle-based belief propagation . . IEEE Trans. Syst. Man Cybern. B , , 2008 . . 38 ( ( 1 ): ): 196 - - 209 . . DOI: 10.1109/TSMCB.2007.910533 http://doi.org/10.1109/TSMCB.2007.910533 . .

Z Zhang . . A flexible new technique for camera calibration . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2000 . . 22 ( ( 11 ): ): 1330 - - 1334 . . DOI: 10.1109/34.888718 http://doi.org/10.1109/34.888718 . .

NN Zheng , , , ZY Liu , , , PJ Ren , , , 等 . . Hybridaugmented intelligence: collaboration and cognition , , 2017 . . : Front. Inform. Technol. Electron. Eng., : in press DOI: 10.1631/FITEE.1700053 http://doi.org/10.1631/FITEE.1700053 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Motion planning of a quadrotor robot game using a simulation-based projected policy iteration method

Safe navigation of quadrotors with jerk limited trajectory

Related Author

Li-dong ZHANG

Ban WANG

Zhi-xiang LIU

You-min ZHANG

Jian-liang AI

Shu-peng LAI

Meng-lu LAN

Ya-xuan LI

Related Institution

Department of Aeronautics and Astronautics, Fudan University

Department of Mechanical, Industrial and Aerospace Engineering, Concordia University, Quebec H3G1M8

Department of Electrical and Computer Engineering, National University of Singapore

Graduate School for Integrative Science & Engineering, National University of Singapore

School of Control Science and Engineering, Shandong University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰