A hybrid visual servo control method for simultaneously controlling a nonholonomic mobile and a manipulator

Wei LI; Rong XIONG

doi:10.1631/FITEE.1900460

Your Location：

Home >

Browse articles >

A hybrid visual servo control method for simultaneously controlling a nonholonomic mobile and a manipulator

Regular Papers | Updated：2022-05-19

- A hybrid visual servo control method for simultaneously controlling a nonholonomic mobile and a manipulator
- 一种同时控制非完整约束底盘和机械臂的混合视觉伺服方法
- Frontiers of Information Technology & Electronic Engineering Vol. 22, Issue 2, Pages: 141-154(2021)
- Affiliations：
  
  State Key Laboratory of Industrial Control Technology, Zhejiang University, Hangzhou 310027, China
- Author bio：
  
  [ "Wei LI, E-mail: li_wei_666@163.com" ]
  Rong XIONG, E-mail: rxiong@zju.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900460
  CLC： TP13
- Received：31 August 2019，
  
  Revised：2020-;5-18，
  
  Published Online：26 June 2020，
  
  Published：2021-02
- Accepted：
Scan QR Code
Wei LI, Rong XIONG. A hybrid visual servo control method for simultaneously controlling a nonholonomic mobile and a manipulator[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(2): 141-154.
DOI：

Wei LI, Rong XIONG. A hybrid visual servo control method for simultaneously controlling a nonholonomic mobile and a manipulator[J]. Frontiers of Information Technology & Electronic Engineering, 2021, 22(2): 141-154. DOI： 10.1631/FITEE.1900460.

摘要

视觉伺服控制规则是指利用机器人上所安装相机获取的图像数据进行机器人运动规划的控制方法，通常分为基于图像的视觉伺服、基于位置的视觉伺服和混合视觉伺服（HVS）控制规则。移动操作可扩大机械臂的工作范围和灵活性，但目前将视觉伺服控制应用于完整移动作业机器人运动的研究较少。本文提出一种面向6自由度机械臂与非完整移动基座组合构成的移动操作机器人HVS运动控制方法。基于移动操作机器人的运动学微分方程，推导了整个机器人的全局雅可比矩阵，进而结合位置信息和视觉图像信息推导了HVS控制方程。夹持器和目标之间的距离是通过安装在夹持器上的摄像机观察标记来计算的，并计算了在图像坐标系中标记特征点位置与期望位置之间的差异。将这些差异代入控制方程，可得到移动操作机器人各自由度的速度控制规律。为避免由观测引起的位置误差，引入卡尔曼滤波来校正机械臂末端的位置和方向。最后，在一台由Bulldog底盘、UR5机械臂和ZED相机组成的移动操作平台上进行了实验验证。

Abstract

Visual servo control rules that refer to the control methods of robot motion planning using image data acquired from the camera mounted on the robot have been widely applied to the motion control of robotic arms or mobile robots. The methods are usually classified as image-based visual servo

position-based visual servo

and hybrid visual servo (HVS) control rules. Mobile manipulation enhances the working range and flexibility of robotic arms. However

there is little work on applying visual servo control rules to the motion of the whole mobile manipulation robot. We propose an HVS motion control method for a mobile manipulation robot which combines a six-degree-of-freedom (6-DOF) robotic arm with a nonholonomic mobile base. Based on the kinematic differential equations of the mobile manipulation robot

the global Jacobian matrix of the whole robot is derived

and the HVS control equation is derived using the whole Jacobian matrix combined with position and visual image information. The distance between the gripper and target is calculated through the observation of the marker by a camera mounted on the gripper. The differences between the positions of the markers' feature points and the expected positions of them in the image coordinate system are also calculated. These differences are substituted into the control equation to obtain the speed control law of each degree of freedom of the mobile manipulation robot. To avoid the position error caused by observation

we also introduce the Kalman filter to correct the positions and orientations of the end of the manipulator. Finally

the proposed algorithm is validated on a mobile manipulation platform consisting of a Bulldog chassis

a UR5 robotic arm

and a ZED camera.

关键词

Keywords

references

DJ Agravante , , , F Chaumette . . Active vision for pose estimation applied to singularity avoidance in visual servoing . . IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2017 . . 2947 - - 2952 . . DOI: 10.1109/IROS.2017.8206129 http://doi.org/10.1109/IROS.2017.8206129 . .

Q Bateux , , , E Marchand , , , J Leitner , , , 等 . . Visual servoing from deep neural networks . . 2017 . . https://arxiv.org/abs/170508940 https://arxiv.org/abs/170508940 , , . .

S Benhimane , , , E Malis . . Homography-based 2D visual tracking and servoing . . Int J Robot Res , , 2007 . . 26 ( ( 7 ): ): 661 - - 676 . . DOI: 10.1177/0278364907080252 http://doi.org/10.1177/0278364907080252 . .

F Chaumette , , , S Hutchinson . . Visual servo control . . I. Basic approaches. IEEE Robot Autom Mag , , 2006 . . 13 ( ( 4 ): ): 82 - - 90 . . DOI: 10.1109/MRA.2006.250573 http://doi.org/10.1109/MRA.2006.250573 . .

F Chaumette , , , S Hutchinson . . Visual servo control . . ò. Advanced approaches. IEEE Robot Autom Mag , , 2007 . . 14 ( ( 1 ): ): 109 - - 118 . . DOI: 10.1109/MRA.2007.339609 http://doi.org/10.1109/MRA.2007.339609 . .

GQ Dong , , , ZH Zhu . . Position-based visual servo control of autonomous robotic manipulators . . Acta Astronaut , , 2015 . . 115 291 - - 302 . . DOI: 10.1016/j.actaastro.2015.05.036 http://doi.org/10.1016/j.actaastro.2015.05.036 . .

C Fantacci , , , G Vezzani , , , U Pattacini , , , 等 . . Markerless visual servoing on unknown objects for humanoid robot platforms . . IEEE Int Conf on Robotics and Automation , , 2018 . . 3099 - - 3106 . . DOI: 10.1109/icra.2018.8462914 http://doi.org/10.1109/icra.2018.8462914 . .

AHA Hafez , , , E Cervera , , , C Jawahar . . Hybrid visual servoing by boosting IBVS and PBVS . . 3 rd Int Conf on Information and Communication Technologies: from Theory to Applications , , 2008 . . 1 - - 6 . . DOI: 10.1109/ICTTA.2008.4530116 http://doi.org/10.1109/ICTTA.2008.4530116 . .

R Horaud , , , B Conio , , , O Leboulleux , , , 等 . . An analytic solution for the perspective 4-point problem . . Comput Vis Graph Image Process , , 1989 . . 47 ( ( 1 ): ): 33 - - 44 . . DOI: 10.1016/0734-189X(89)90052-2 http://doi.org/10.1016/0734-189X(89)90052-2 . .

G Hu , , , W MacKunis , , , N Gans , , , 等 . . Homography-based visual servo control via an uncalibrated camera . . American Control Conf , , 2008 . . 4791 - - 4796 . . DOI: 10.1109/ACC.2008.4587252 http://doi.org/10.1109/ACC.2008.4587252 . .

O Kermorgant , , , F Chaumette . . Combining IBVS and PBVS to ensure the visibility constraint . . IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2011 . . 2849 - - 2854 . . DOI: 10.1109/IROS.2011.6094589 http://doi.org/10.1109/IROS.2011.6094589 . .

HX Lang , , , MT Khan , , , KK Tan , , , 等 . . Application of visual servo control in autonomous mobile rescue robots . . Int J Comput Commun Contr , , 2016 . . 11 ( ( 5 ): ): 685 - - 696 . . DOI: 10.15837/ijccc.2016.5.2680 http://doi.org/10.15837/ijccc.2016.5.2680 . .

S Levine , , , P Pastor , , , A Krizhevsky , , , 等 . . Learning hand-eye coordination for robotic grasping with large-scale data collection . . Int Symp on Experimental Robotics , , 2016 . . 173 - - 184 . . DOI: 10.1007/978-3-319-50115-4_16 http://doi.org/10.1007/978-3-319-50115-4_16 . .

T Sandy , , , J Buchli . . Dynamically decoupling base and end-effector motion for mobile manipulation using visual-inertial sensing . . IEEE/RSJ Int Conf on Intelligent Robots and Systems , , 2017 . . 6299 - - 6306 . . DOI: 10.1109/IROS.2017.8206533 http://doi.org/10.1109/IROS.2017.8206533 . .

DP Tsakiris , , , P Rives , , , C Samson . . Extending visual servoing techniques to nonholonomic mobile robots . . In: Kriegman DJ, Hager GD, Morse AS (Eds.), The Confluence of Vision and Control. Springer, London , , 1998 . . 106 - - 117 . . DOI: 10.1007/BFb0109666 http://doi.org/10.1007/BFb0109666 . .

Y Wang , , , GL Zhang , , , HX Lang , , , 等 . . A modified image-based visual servo controller with hybrid camera configuration for robust robotic grasping . . Robot Auton Syst , , 2014 . . 62 ( ( 10 ): ): 1398 - - 1407 . . DOI: 10.1016/j.robot.2014.06.003 http://doi.org/10.1016/j.robot.2014.06.003 . .

XG Zhong , , , XY Zhong , , , XF Peng . . Robots visual servo control with features constraint employing Kalman-neural-network filtering scheme . . Neurocomputing , , 2015 . . 151 268 - - 277 . . DOI: 10.1016/j.neucom.2014.09.043 http://doi.org/10.1016/j.neucom.2014.09.043 . .

Views

124

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Approximate Gaussian conjugacy: parametric recursive filtering under nonlinearity, multimodality, uncertainty, and constraint, and beyond

Related Author

LI Tian-cheng

SU Jin-ya

LIU Wei

CORCHADO Juan M.

Related Institution

School of Sciences, University of Salamanca

Department of Aeronautical and Automotive Engineering, Loughborough University, Loughborough

Department of Electronic and Electrical Engineering, University of Sheffield

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰