Autonomous flying blimp interaction with human in an indoor space

Ning-shi YAO; Qiu-yang TAO; Wei-yu LIU; Zhen LIU; Ye TIAN; Pei-yu WANG; Timothy LI; Fumin ZHANG

doi:10.1631/FITEE.1800587

Your Location：

Home >

Browse articles >

Autonomous flying blimp interaction with human in an indoor space

Special Issue on Complex System and Intelligent Control | Updated：2022-05-19

- Autonomous flying blimp interaction with human in an indoor space
- 气球机器人在室内环境下与人的自然交互
- Frontiers of Information Technology & Electronic Engineering Vol. 20, Issue 1, Pages: 45-59(2019)
- Affiliations：
  
  School of Electrical and Computer Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
  College of Computing, Georgia Institute of Technology, Atlanta, GA 30332, USA
- Author bio：
  
  [ "Ning-shi YAO, E-mail: nyao6@gatech.edu" ]
  [ "Qiu-yang TAO, E-mail: qtao7@gatech.edu" ]
  [ "Wei-yu LIU, E-mail: wliu88@gatech.edu" ]
  Fumin ZHANG, E-mail: fumin@gatech.edu
- Funds：
- DOI：10.1631/FITEE.1800587
  CLC： TP24
- Received：20 September 2018，
  
  Revised：2019-;1-;8，
  
  Published：2019-01
- Accepted：
Scan QR Code
Ning-shi YAO, Qiu-yang TAO, Wei-yu LIU, et al. Autonomous flying blimp interaction with human in an indoor space[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(1): 45-59.
DOI：

Ning-shi YAO, Qiu-yang TAO, Wei-yu LIU, et al. Autonomous flying blimp interaction with human in an indoor space[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(1): 45-59. DOI： 10.1631/FITEE.1800587.

摘要

为支持室内环境下长达两小时及以上的人机交互实验，我们研发了一款“佐治亚理工全自动迷你气球飞行机器人”（GT-MAB）。GT-MAB可以安全地与人近距离互动。该机器人具有人脸识别、自动跟踪人类使用者以及手势识别等功能。采用深度人工神经网络检测GT-MAB实时视频信号中人类使用者的脸和手，并基于此功能，设计一套人机交互系统实现GT-MAB与使用者之间的自然交互。GT-MAB的学习算法可以自主识别两种手势，因此在交互过程中，使用者无需佩戴任何装置就可与GT-MAB互动。基于机器视觉的反馈控制器可控制GT-MAB自动追踪并跟随使用者。一旦机器人成功识别使用者的手势，机器人会以对应的飞行轨迹回应使用者手势，同时将识别结果以及意图通过其自身搭载的LED显示屏展示给使用者。实验数据表明，向人类使用者展示的视觉信号能明显提高人类使用者和GT-MAB之间的交互体验。实验结果表明，GT-MAB作为一款新型飞行机器人，可在室内环境下安全地与人互动并有效采集人机交互数据。

Abstract

We present the Georgia Tech Miniature Autonomous Blimp (GT-MAB)

which is designed to support human-robot interaction experiments in an indoor space for up to two hours. GT-MAB is safe while flying in close proximity to humans. It is able to detect the face of a human subject

follow the human

and recognize hand gestures. GT-MAB employs a deep neural network based on the single shot multibox detector to jointly detect a human user's face and hands in a real-time video stream collected by the onboard camera. A human-robot interaction procedure is designed and tested with various human users. The learning algorithms recognize two hand waving gestures. The human user does not need to wear any additional tracking device when interacting with the flying blimp. Vision-based feedback controllers are designed to control the blimp to follow the human and fly in one of two distinguishable patterns in response to each of the two hand gestures. The blimp communicates its intentions to the human user by displaying visual symbols. The collected experimental data show that the visual feedback from the blimp in reaction to the human user significantly improves the interactive experience between blimp and human. The demonstrated success of this procedure indicates that GT-MAB could serve as a flying robot that is able to collect human data safely in an indoor environment.

关键词

Keywords

references

U Acharya , , , A Bevins , , , BA Duncan . . Investigation of human-robot comfort with a small unmanned aerial vehicle compared to a ground robot . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2017 . . p.2758 - - 2765 . . DOI: 10.1109/IROS.2017.8206104 http://doi.org/10.1109/IROS.2017.8206104 . .

D Arroyo , , , C Lucho , , , J Roncal , , , 等 . . Daedalus: a sUAV for human-robot interaction . . Proc 9 th ACM/IEEE Int Conf on Human-Robot Interaction , , 2014 . . p.116 - - 117 . . . .

S Birchfield . . KLT: . . an Implementation of the Kanade- Lucas-Tomasi Feature Tracker , , 1996 . . .

M Burri , , , L Gasser , , , M Käch , , , 等 . . Design and control of a spherical omnidirectional blimp . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2013 . . p.1873 - - 1879 . . DOI: 10.1109/IROS.2013.6696604 http://doi.org/10.1109/IROS.2013.6696604 . .

JR Cauchard , , , KY Zhai , , , M Spadafora , , , 等 . . Emotion encoding in human-drone interaction . . Proc 11 th ACM/IEEE Int Conf on Human-Robot Interaction , , 2016 . . p.263 - - 270 . . DOI: 10.1109/HRI.2016.7451761 http://doi.org/10.1109/HRI.2016.7451761 . .

S Cho , , , V Mishra , , , Q Tao , , , 等 . . Autopilot design for a class of miniature autonomous blimps . . Proc IEEE Conf on Control Technology and Applications , , 2017 . . p.841 - - 846 . . DOI: 10.1109/CCTA.2017.8062564 http://doi.org/10.1109/CCTA.2017.8062564 . .

P Corke . . Robotics, Vision and Control: Fundamental Algorithms in MATLAB . . Springer Berlin Germany , , 2011 . . .

G Costante , , , E Bellocchio , , , P Valigi , , , 等 . . Personalizing vision-based gestural interfaces for HRI with UAVs: a transfer learning approach . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2014 . . p.3319 - - 3326 . . DOI: 10.1109/IROS.2014.6943024 http://doi.org/10.1109/IROS.2014.6943024 . .

F de Crescenzio , , , G Miranda , , , F Persiani , , , 等 . . A first implementation of an advanced 3D interface to control and supervise UAV (uninhabited aerial vehicles) missions . . Presence , , 2009 . . 18 ( ( 3 ): ): 171 - - 184 . . DOI: 10.1162/pres.18.3.171 http://doi.org/10.1162/pres.18.3.171 . .

M Draper , , , G Calhoun , , , H Ruff , , , 等 . . Manual versus speech input for unmanned aerial vehicle control station operations . . Proc Hum Factors Ergon Soc Ann Meet , , 2003 . . 47 ( ( 1 ): ): 109 - - 113 . . DOI: 10.1177/154193120304700123 http://doi.org/10.1177/154193120304700123 . .

BR Duffy . . Anthropomorphism and the social robot . . Rob Auton Syst , , 2003 . . 42 ( ( 3-4 ): ): 177 - - 190 . . DOI: 10.1016/S0921-8890(02)00374-3 http://doi.org/10.1016/S0921-8890(02)00374-3 . .

BA Duncan , , , RR Murphy . . Comfortable approach distance with small unmanned aerial vehicles . . Proc IEEE RO-MAN , , 2013 . . p.786 - - 792 . . DOI: 10.1109/ROMAN.2013.6628409 http://doi.org/10.1109/ROMAN.2013.6628409 . .

MA Goodrich , , , AC Schultz . . Human-robot interaction: a survey . . Found Trends Hum-Comput Interact , , 2007 . . 1 ( ( 3 ): ): 203 - - 275 . . DOI: 10.1561/1100000005 http://doi.org/10.1561/1100000005 . .

E Graether , , , F Mueller . . Joggobot: a flying robot as jogging companion . . Proc ACM SIGCHI Conf on Human Factors in Computing Systems , , 2012 . . p.1063 - - 1066 . . DOI: 10.1145/2212776.2212386 http://doi.org/10.1145/2212776.2212386 . .

ET Hall . . The Hidden Dimension , , Doubleday, New York, USA , , 1966 . . .

JP Hansen , , , A Alapetite , , , IS MacKenzie , , , 等 . . The use of gaze to control drones . . Proc Symp on Eye Tracking Research and Applications , , 2014 . . p.27 - - 34 . . DOI: 10.1145/2578153.2578156 http://doi.org/10.1145/2578153.2578156 . .

D He , , , HY Ren , , , WD Hua , , , 等 . . Flyingbuddy: augment human mobility and perceptibility . . Proc 13 th Int Conf on Ubiquitous Computing , , 2011 . . p.615 - - 616 . . DOI: 10.1145/2030112.2030241 http://doi.org/10.1145/2030112.2030241 . .

D Helbing , , , P Molnár . . Social force model for pedestrian dynamics . . Phys Rev E , , 1995 . . 51 ( ( 5 ): ): 4282 - - 4286 . . DOI: 10.1103/PhysRevE.51.4282 http://doi.org/10.1103/PhysRevE.51.4282 . .

M Lichtenstern , , , M Frassl , , , B Perun , , , 等 . . A prototyping environment for interaction between a human and a robotic multi-agent system . . Proc 7 th ACM/IEEE Int Conf on Human-Robot Interaction , , 2012 . . p.185 - - 186 . . DOI: 10.1145/2157689.2157747 http://doi.org/10.1145/2157689.2157747 . .

CF Liew , , , T Yairi . . Quadrotor or blimp? Noise and appearance considerations in designing social aerial robot . . Proc 8 th ACM/IEEE Int Conf on Human-Robot Interaction , , 2013 . . p.183 - - 184 . . DOI: 10.1109/HRI.2013.6483562 http://doi.org/10.1109/HRI.2013.6483562 . .

H Lim , , , SN Sinha . . Monocular localization of a moving person onboard a quadrotor MAV . . Proc IEEE Int Conf on Robotics and Automation , , 2015 . . p.2182 - - 2189 . . DOI: 10.1109/ICRA.2015.7139487 http://doi.org/10.1109/ICRA.2015.7139487 . .

W Liu , , , D Anguelov , , , D Erhan , , , 等 . . SSD: single shot multibox detector . . Proc 14 th European Conf on Computer Vision , , 2016 . . p.21 - - 37 . . DOI: 10.1007/978-3-319-46448-0_2 http://doi.org/10.1007/978-3-319-46448-0_2 . .

A Mittal , , , A Zisserman , , , PHS Torr . . Hand detection using multiple proposals . . Proc British Machine Vision Conf , , 2011 . . p.1 - - 11 . . . .

VM Monajjemi , , , J Wawerla , , , R Vaughan , , , 等 . . HRI in the sky: creating and commanding teams of UAVs with a vision-mediated gestural interface . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2013 . . p.617 - - 623 . . DOI: 10.1109/IROS.2013.6696415 http://doi.org/10.1109/IROS.2013.6696415 . .

VM Monajjemi , , , S Mohaimenianpour , , , R Vaughan . . UAV, come to me: end-to-end, multi-scale situated HRI with an uninstrumented human and a distant UAV . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2016 . . p.4410 - - 4417 . . DOI: 10.1109/IROS.2016.7759649 http://doi.org/10.1109/IROS.2016.7759649 . .

J Nagi , , , A Giusti , , , GA di Caro , , , 等 . . Human control of UAVs using face pose estimates and hand gestures . . Proc ACM/IEEE Int Conf on Human-Robot Interaction , , 2014 . . p.252 - - 253 . . DOI: 10.1145/2559636.2559833 http://doi.org/10.1145/2559636.2559833 . .

T Naseer , , , J Sturm , , , D Cremers . . FollowMe: person following and gesture recognition with a quadrocopter . . Proc IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2013 . . p.624 - - 630 . . DOI: 10.1109/IROS.2013.6696416 http://doi.org/10.1109/IROS.2013.6696416 . .

AG Perera , , , YW Law , , , J Chahl . . Human pose and path estimation from aerial video using dynamic classifier selection . . Cogn Comput , , 2018 . . 6 ( ( 10 ): ): 1019 - - 1041 . . DOI: 10.1007/s12559-018-9577-6 http://doi.org/10.1007/s12559-018-9577-6 . .

E Peshkova , , , M Hitz , , , B Kaufmann . . Natural interaction techniques for an unmanned aerial vehicle system . . IEEE Perv Comput , , 2017 . . 16 ( ( 1 ): ): 34 - - 42 . . DOI: 10.1109/MPRV.2017.3 http://doi.org/10.1109/MPRV.2017.3 . .

S Pourmehr , , , VM Monajjemi , , , SA Sadat , , , 等 . . "You are green": a touch-to-name interaction in an integrated multi-modal multi-robot HRI system . . Proc ACM/IEEE Int Conf on Human-Robot Interaction , , 2014 . . p.266 - - 267 . . DOI: 10.1145/2559636.2559806 http://doi.org/10.1145/2559636.2559806 . .

S Schneegass , , , F Alt , , , J Scheible , , , 等 . . Midair displays: concept and first experiences with free-floating pervasive displays . . Proc Int Symp on Pervasive Displays, Article 27 , , 2014 . . DOI: 10.1145/2611009.2611013 http://doi.org/10.1145/2611009.2611013 . .

M Sharma , , , D Hildebrandt , , , G Newman , , , 等 . . Communicating affect via flight path: exploring use of the Laban effort system for designing affective locomotion paths . . Proc ACM/IEEE Int Conf on Human-Robot Interaction , , 2013 . . p.293 - - 300 . . DOI: 10.1109/HRI.2013.6483602 http://doi.org/10.1109/HRI.2013.6483602 . .

V Srisamosorn , , , N Kuwahara , , , A Yamashita , , , 等 . . Design of face tracking system using fixed 360-degree cameras and flying blimp for health care evaluation . . Proc 4 th Int Conf on Serviceology , , 2016 . . .

D St-Onge , , , PY Brèches , , , I Sharf , , , 等 . . Control, localization and human interaction with an autonomous lighter-than-air performer . . Rob Auton Syst , , 2017 . . 88 165 - - 186 . . DOI: 10.1016/j.robot.2016.10.013 http://doi.org/10.1016/j.robot.2016.10.013 . .

D Szafir , , , B Mutlu , , , T Fong . . Communication of intent in assistive free flyers . . Proc ACM/IEEE Int Conf on Human-Robot Interaction , , 2014 . . p.358 - - 365 . . DOI: 10.1145/2559636.2559672 http://doi.org/10.1145/2559636.2559672 . .

D Szafir , , , B Mutlu , , , T Fong . . Communicating directionality in flying robots . . Proc 10 th Annual ACM/IEEE Int Conf on Human-Robot Interaction , , 2015 . . p.19 - - 26 . . DOI: 10.1145/2696454.2696475 http://doi.org/10.1145/2696454.2696475 . .

QY Tao , , , J Cha , , , MX Hou , , , 等 . . Parameter identification of blimp dynamics through swinging motion . . Proc 15 th Int Conf on Control, Automation, Robotics and Vision , , 2018 . . DOI: 10.1109/ICARCV.2018.8581376 http://doi.org/10.1109/ICARCV.2018.8581376 . .

P Viola , , , MJ Jones . . Robust real-time face detection . . Int J Comput Vis , , 2004 . . 57 ( ( 2 ): ): 137 - - 154 . . DOI: 10.1023/B:VISI.0000013087.49260.fb http://doi.org/10.1023/B:VISI.0000013087.49260.fb . .

S Wold , , , K Esbensen , , , P Geladi . . Principal component analysis . . Chemom Intell Lab Syst , , 1987 . . 2 ( ( 1-3 ): ): 37 - - 52 . . DOI: 10.1016/0169-7439(87)80084-9 http://doi.org/10.1016/0169-7439(87)80084-9 . .

NS Yao , , , E Anaya , , , QY Tao , , , 等 . . Monocular visionbased human following on miniature robotic blimp . . Proc IEEE Int Conf on Robotics and Automation , , 2017 . . p.3244 - - 3249 . . DOI: 10.1109/ICRA.2017.7989369 http://doi.org/10.1109/ICRA.2017.7989369 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems

Related Author

Yusong ZHOU

Xiaoyu JIANG

Shu SUN

Xinmin ZHANG

Yuanqiu MO

Zhihuan SONG

Maokun ZHENG

Zhi LI

Related Institution

School of Mathematics, Southeast University

State Key Laboratory of Industrial Control Technology, Zhejiang University

College of Control Science and Engineering, Zhejiang University

State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University

School of Electronic and Electrical Engineering, Shanghai University of Engineering Science

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰