A quantitative attribute-based benchmark methodology for single-target visual tracking

KANG Wen-jing; LIU Chang; LIU Gong-liang

doi:10.1631/FITEE.1900245

Your Location：

Home >

Browse articles >

A quantitative attribute-based benchmark methodology for single-target visual tracking

Regular Papers | Updated：2022-05-19

- A quantitative attribute-based benchmark methodology for single-target visual tracking
- 基于定量属性的单目标视觉跟踪算法评价体系研究
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 3, Pages: 405-421(2020)
- Affiliations：
  
  School of Information Science and Engineering, Harbin Institute of Technology, Weihai 264209, China
  School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore 639798, Singapore
- Author bio：
  
  [ "Wen-jing KANG, E-mail: kwjqq@hit.edu.cn" ]
  [ "Chang LIU, E-mail: liuc0051@e.ntu.edu.sg" ]
  Gong-liang LIU, E-mail: liugl@hit.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900245
  CLC： TP723
- Received：15 May 2019，
  
  Revised：2019-11-12，
  
  Published：2020-03
- Accepted：
Scan QR Code
KANG Wen-jing, LIU Chang, LIU Gong-liang. A quantitative attribute-based benchmark methodology for single-target visual tracking[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(3): 405-421.
DOI：

KANG Wen-jing, LIU Chang, LIU Gong-liang. A quantitative attribute-based benchmark methodology for single-target visual tracking[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(3): 405-421. DOI： 10.1631/FITEE.1900245.

摘要

视觉跟踪是计算机视觉领域热门研究课题之一。近年来，很多先进跟踪算法和性能评价基准相继发布，并取得巨大成功。现有评价体系大多定位于衡量整体性能，无法通过针对性的详细论证评估跟踪器的优势和缺点，且很多常用评测指标缺乏令人信服的含义解释。本文从测试数据、测试方法、测试指标3方面深入分析跟踪评价体系的细节。首先，归纳整理了12个反映图像序列不同特性的帧间视觉属性，并首次定量给出其归一化公式。基于这些属性定义，提出两种新的测试方法，即基于相关性的测试和基于权重的测试，使评价体系能更直观、更清晰地评定跟踪器各方面性能。然后，将所提测试方法应用于著名的跟踪挑战赛，即Video Object Tracking (VOT) Challenge 2017。测试结果表明，在目标尺寸快速或剧烈变化时，跟踪器大多表现不佳，即使基于深度学习的先进跟踪器也未能很好解决这一问题。此外发现，中心位置差错（center location error，CLE）性能指标虽未考虑到目标尺度，在实际测试中仍对目标尺寸变化很敏感。

Abstract

In the past several years

various visual object tracking benchmarks have been proposed

and some of them have been used widely in numerous recently proposed trackers. However

most of the discussions focus on the overall performance

and cannot describe the strengths and weaknesses of the trackers in detail. Meanwhile

several benchmark measures that are often used in tests lack convincing interpretation. In this paper

12 frame-wise visual attributes that reflect different aspects of the characteristics of image sequences are collated

and a normalized quantitative formulaic definition has been given to each of them for the first time. Based on these definitions

we propose two novel test methodologies

a correlation-based test and a weight-based test

which can provide a more intuitive and easier demonstration of the trackers' performance for each aspect. Then these methods have been applied to the raw results from one of the most famous tracking challenges

the Video Object Tracking (VOT) Challenge 2017. From the tests

most trackers did not perform well when the size of the target changed rapidly or intensely

and even the advanced deep learning based trackers did not perfectly solve the problem. The scale of the targets was not considered in the calculation of the center location error; however

in a practical test

the center location error is still sensitive to the targets' changes in size.

关键词

Keywords

references

B Babenko , , , MH Yang , , , S Belongie . . Robust object tracking with online multiple instance learning . . IEEE Trans Patt Anal Mach Intell , , 2011 . . 33 ( ( 8 ): ): 1619 - - 1632 . . DOI: 10.1109/TPAMI.2010.226 http://doi.org/10.1109/TPAMI.2010.226 . .

CL Bao , , , Y Wu , , , HB Ling , , , 等 . . Real time robust L1 tracker using accelerated proximal gradient approach . . IEEE Conf on Computer Vision and Pattern Recognition , , 2012 . . p.1830 - - 1837 . . DOI: 10.1109/CVPR.2012.6247881 http://doi.org/10.1109/CVPR.2012.6247881 . .

F Battistone , , , A Petrosino , , , V Santopietro . . Watch out: embedded video tracking with BST for unmanned aerial vehicles . . J Signal Process Syst , , 2018 . . 90 ( ( 6 ): ): 891 - - 900 . . DOI: 10.1007/s11265-017-1279-x http://doi.org/10.1007/s11265-017-1279-x . .

L Bertinetto , , , J Valmadre , , , S Golodetz , , , 等 . . Staple: complementary learners for real-time tracking . . IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.1401 - - 1409 . . DOI: 10.1109/CVPR.2016.156 http://doi.org/10.1109/CVPR.2016.156 . .

L Čehovin , , , M Kristan , , , A Leonardis . . An adaptive coupled-layer visual model for robust visual tracking . . IEEE Int Conf on Computer Vision , , 2011 . . p.1363 - - 1370 . . DOI: 10.1109/ICCV.2011.6126390 http://doi.org/10.1109/ICCV.2011.6126390 . .

L Čehovin , , , A Leonardis , , , M Kristan . . Visual object tracking performance measures revisited . . IEEE Trans Image Process , , 2016a . . 25 ( ( 3 ): ): 1261 - - 1274 . . DOI: 10.1109/TIP.2016.2520370 http://doi.org/10.1109/TIP.2016.2520370 . .

L Čehovin , , , A Leonardis , , , M Kristan . . Robust visual tracking using template anchors . . IEEE Winter Conf on Applications of Computer Vision , , 2016b . . p.1 - - 8 . . DOI: 10.1109/WACV.2016.7477570 http://doi.org/10.1109/WACV.2016.7477570 . .

K Chen , , , WB Tao . . Convolutional regression for visual tracking . . IEEE Trans Image Process , , 2018 . . 27 ( ( 7 ): ): 3611 - - 3620 . . DOI: 10.1109/TIP.2018.2819362 http://doi.org/10.1109/TIP.2018.2819362 . .

R Collins , , , XH Zhou , , , SK Teh . . An open source tracking testbed and evaluation web site . . Proc IEEE Int Workshop on Performance Evaluation of Tracking and Surveillance , , 2005 . . p.17 - - 24 . . . .

N Dalal , , , B Triggs . . Histograms of oriented gradients for human detection . . IEEE Conf on Computer Vision and Pattern Recognition , , 2005 . . p.886 - - 893 . . DOI: 10.1109/CVPR.2005.177 http://doi.org/10.1109/CVPR.2005.177 . .

M Danelljan , , , G Häger , , , F Khan , , , 等 . . Accurate scale estimation for robust visual tracking . . Proc British Machine Vision Conf , , 2014 . . p.1 - - 11 . . DOI: 10.5244/C.28.65 http://doi.org/10.5244/C.28.65 . .

M Danelljan , , , G Häger , , , FS Khan , , , 等 . . Convolutional features for correlation filter based visual tracking . . Proc IEEE Int Conf on Computer Vision Workshops , , 2015a . . p.621 - - 629 . . DOI: 10.1109/ICCVW.2015.84 http://doi.org/10.1109/ICCVW.2015.84 . .

M Danelljan , , , G Häger , , , FS Khan , , , 等 . . Learning spatially regularized correlation filters for visual tracking . . IEEE Int Conf on Computer Vision , , 2015b . . p.4310 - - 4318 . . DOI: 10.1109/ICCV.2015.490 http://doi.org/10.1109/ICCV.2015.490 . .

M Danelljan , , , A Robinson , , , FS Khan , , , 等 . . Beyond correlation filters: learning continuous convolution operators for visual tracking . . 14 th European Conf on Computer Vision , , 2016 . . p.472 - - 488 . . DOI: 10.1007/978-3-319-46454-1_29 http://doi.org/10.1007/978-3-319-46454-1_29 . .

M Danelljan , , , G Bhat , , , FS Khan , , , 等 . . ECO: efficient convolution operators for tracking . . IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.6931 - - 6939 . . DOI: 10.1109/CVPR.2017.733 http://doi.org/10.1109/CVPR.2017.733 . .

MA Fischler , , , RC Bolles . . Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography . . Commun ACM , , 1981 . . 24 ( ( 6 ): ): 381 - - 395 . . DOI: 10.1145/358669.358692 http://doi.org/10.1145/358669.358692 . .

B Funt , , , K Barnard , , , L Martin . . Is machine colour constancy good enough? . . 5 th European Conf on Computer Vision , , 1998 . . p.445 - - 459 . . DOI: 10.1007/BFb0055683 http://doi.org/10.1007/BFb0055683 . .

HK Galoogahi , , , A Fagg , , , C Huang , , , 等 . . Need for speed: a benchmark for higher frame rate object tracking . . IEEE Int Conf on Computer Vision , , 2017 . . p.1134 - - 1143 . . DOI: 10.1109/ICCV.2017.128 http://doi.org/10.1109/ICCV.2017.128 . .

SB Gao , , , KF Yang , , , CY Li , , , 等 . . Color constancy using double-opponency . . IEEE Trans Patt Anal Mach Intell , , 2015 . . 37 ( ( 10 ): ): 1973 - - 1985 . . DOI: 10.1109/TPAMI.2015.2396053 http://doi.org/10.1109/TPAMI.2015.2396053 . .

E Gundogdu , , , AA Alatan . . Good features to correlate for visual tracking . . IEEE Trans Image Process , , 2018 . . 27 ( ( 5 ): ): 2526 - - 2540 . . DOI: 10.1109/TIP.2018.2806280 http://doi.org/10.1109/TIP.2018.2806280 . .

S Hare , , , A Saffari , , , PHS Torr . . Struck: structured output tracking with kernels . . Int Conf on Computer Vision , , 2011 . . p.263 - - 270 . . DOI: 10.1109/ICCV.2011.6126251 http://doi.org/10.1109/ICCV.2011.6126251 . .

S Hare , , , S Golodetz , , , A Saffari , , , 等 . . Struck: structured output tracking with kernels . . IEEE Trans Patt Anal Mach Intell , , 2016 . . 38 ( ( 10 ): ): 2096 - - 2109 . . DOI: 10.1109/TPAMI.2015.2509974 http://doi.org/10.1109/TPAMI.2015.2509974 . .

Z He , , , Y Fan , , , J Zhuang , , , 等 . . Correlation filters with weighted convolution responses . . IEEE Int Conf on Computer Vision Workshop , , 2017 . . p.1992 - - 2000 . . . .

JF Henriques , , , R Caseiro , , , P Martins , , , 等 . . High-speed tracking with kernelized correlation filters . . IEEE Trans Patt Anal Mach Intell , , 2015 . . 37 ( ( 3 ): ): 583 - - 596 . . DOI: 10.1109/TPAMI.2014.2345390 http://doi.org/10.1109/TPAMI.2014.2345390 . .

B Karasulu , , , S Korukoglu . . A software for performance evaluation and comparison of people detection and tracking methods in video processing . . Multim Tools Appl , , 2011 . . 55 ( ( 3 ): ): 677 - - 723 . . DOI: 10.1007/s11042-010-0591-2 http://doi.org/10.1007/s11042-010-0591-2 . .

M Kristan , , , J Perš , , , M Perše , , , 等 . . A Bayes-spectralentropy-based measure of camera focus using a discrete cosine transform . . Patt Recogn Lett , , 2006 . . 27 ( ( 13 ): ): 1431 - - 1439 . . DOI: 10.1016/j.patrec.2006.01.016 http://doi.org/10.1016/j.patrec.2006.01.016 . .

M Kristan , , , R Pflugfelder , , , A Leonardis , , , 等 . . The Visual Object Tracking VOT2013 Challenge results . . IEEE Int Conf on Computer Vision Workshops , , 2013 . . p.98 - - 111 . . DOI: 10.1109/ICCVW.2013.20 http://doi.org/10.1109/ICCVW.2013.20 . .

M Kristan , , , R Pflugfelder , , , A Leonardis , , , 等 . . The Visual Object Tracking VOT2014 Challenge results . . European Conf on Computer Vision , , 2015a . . p.191 - - 217 . . DOI: 10.1007/978-3-319-16181-5_14 http://doi.org/10.1007/978-3-319-16181-5_14 . .

M Kristan , , , J Matas , , , A Leonardis , , , 等 . . The Visual Object Tracking VOT2015 Challenge results . . IEEE Int Conf on Computer Vision Workshop , , 2015b . . p.564 - - 586 . . DOI: 10.1109/ICCVW.2015.79 http://doi.org/10.1109/ICCVW.2015.79 . .

M Kristan , , , J Matas , , , A Leonardis , , , 等 . . A novel performance evaluation methodology for single-target trackers . . IEEE Trans Patt Anal Mach Intell , , 2016a . . 38 ( ( 11 ): ): 2137 - - 2155 . . DOI: 10.1109/TPAMI.2016.2516982 http://doi.org/10.1109/TPAMI.2016.2516982 . .

M Kristan , , , A Leonardis , , , J Matas , , , 等 . . The Visual Object Tracking VOT2016 Challenge results . . European Conf on Computer Vision , , 2016b . . p.777 - - 823 . . DOI: 10.1007/978-3-319-48881-3_54 http://doi.org/10.1007/978-3-319-48881-3_54 . .

M Kristan , , , A Leonardis , , , J Matas , , , 等 . . The Visual Object Tracking VOT2017 Challenge results . . IEEE Int Conf on Computer Vision Workshops , , 2017 . . p.1949 - - 1972 . . DOI: 10.1109/ICCVW.2017.230 http://doi.org/10.1109/ICCVW.2017.230 . .

J Kwon , , , KM Lee . . Tracking of abrupt motion using Wang-Landau Monte Carlo estimation . . 10 th European Conf on Computer Vision , , 2008 . . p.387 - - 400 . . DOI: 10.1007/978-3-540-88682-2_30 http://doi.org/10.1007/978-3-540-88682-2_30 . .

AN Li , , , M Lin , , , Y Wu , , , 等 . . NUS-PRO: a new visual tracking challenge . . IEEE Trans Patt Anal Mach Intell , , 2016 . . 38 ( ( 2 ): ): 335 - - 349 . . DOI: 10.1109/TPAMI.2015.2417577 http://doi.org/10.1109/TPAMI.2015.2417577 . .

B Li , , , W Wu , , , Q Wang , , , 等 . . SiamRPN++: evolution of Siamese visual tracking with very deep networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2019 . . p.4282 - - 4291 . . . .

SY Li , , , DY Yeung . . Visual object tracking for unmanned aerial vehicles: a benchmark and new motion models . . Proc 31 st AAAI Conf on Artificial Intelligence , , 2017 . . p.4140 - - 4146 . . . .

DG Lowe . . Object recognition from local scale-invariant features . . Proc 7 th IEEE Int Conf on Computer Vision , , 1999 . . p.150 - - 1157 . . DOI: 10.1109/ICCV.1999.790410 http://doi.org/10.1109/ICCV.1999.790410 . .

A Lukežič , , , T Vojír , , , LC Zajc , , , 等 . . Discriminative correlation filter with channel and spatial reliability . . IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.4847 - - 4856 . . DOI: 10.1109/CVPR.2017.515 http://doi.org/10.1109/CVPR.2017.515 . .

A Lukežič , , , LČ Zajc , , , M Kristan . . Deformable parts correlation filters for robust visual tracking . . IEEE Trans Cybern , , 2018 . . 48 ( ( 6 ): ): 1849 - - 1861 . . DOI: 10.1109/TCYB.2017.2716101 http://doi.org/10.1109/TCYB.2017.2716101 . .

R Mathew , , , SS Hiremath . . Control of velocityconstrained stepper motor-driven Hilare robot for waypoint navigation . . Engineering , , 2018 . . 4 ( ( 4 ): ): 491 - - 499 . . DOI: 10.1016/j.eng.2018.07.013 http://doi.org/10.1016/j.eng.2018.07.013 . .

B Mocanu , , , R Tapu , , , T Zaharia . . Single object tracking using offline trained deep regression networks . . 7 th Int Conf on Image Processing Theory, Tools and Applications , , 2017 . . p.1 - - 6 . . DOI: 10.1109/IPTA.2017.8310091 http://doi.org/10.1109/IPTA.2017.8310091 . .

G Nebehay , , , R Pflugfelder . . Clustering of static-adaptive correspondences for deformable object tracking . . IEEE Conf on Computer Vision and Pattern Recognition , , 2015 . . p.2784 - - 2791 . . DOI: 10.1109/CVPR.2015.7298895 http://doi.org/10.1109/CVPR.2015.7298895 . .

DA Ross , , , J Lim , , , RS Lin , , , 等 . . Incremental learning for robust visual tracking . . Int J Comput Vis , , 2008 . . 77 ( ( 1-3 ): ): 125 - - 141 . . DOI: 10.1007/s11263-007-0075-7 http://doi.org/10.1007/s11263-007-0075-7 . .

P Senna , , , IN Drummond , , , GS Bastos . . Real-time ensemble-based tracker with Kalman filter . . 30 th SIBGRAPI Conf on Graphics, Patterns and Images , , 2017 . . p.338 - - 344 . . DOI: 10.1109/SIBGRAPI.2017.51 http://doi.org/10.1109/SIBGRAPI.2017.51 . .

AWM Smeulders , , , DM Chu , , , R Cucchiara , , , 等 . . Visual tracking: an experimental survey . . IEEE Trans Patt Anal Mach Intell , , 2014 . . 36 ( ( 7 ): ): 1442 - - 1468 . . DOI: 10.1109/TPAMI.2013.230 http://doi.org/10.1109/TPAMI.2013.230 . .

C Sun , , , D Wang , , , HC Lu , , , 等 . . Learning spatial-aware regressions for visual tracking . . IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.8962 - - 8970 . . DOI: 10.1109/CVPR.2018.00934 http://doi.org/10.1109/CVPR.2018.00934 . .

A Tran , , , A Manzanera . . Mixing Hough and color histogram models for accurate real-time object tracking . . 17 th Int Conf on Computer Analysis of Images and Patterns , , 2017 . . p.43 - - 54 . . DOI: 10.1007/978-3-319-64689-3_4 http://doi.org/10.1007/978-3-319-64689-3_4 . .

J Valmadre , , , L Bertinetto , , , J Henriques , , , 等 . . End-to-end representation learning for correlation filter based tracking . . IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.5000 - - 5008 . . DOI: 10.1109/CVPR.2017.531 http://doi.org/10.1109/CVPR.2017.531 . .

T Vojíř , , , J Matas . . The enhanced flock of trackers . . In: Cipolla R, Battiato S, Farinella GM (Eds.), Registration and Recognition in Images and Videos. Springer, Berlin , , 2014 . . p.113 - - 136 . . DOI: 10.1007/978-3-642-44907-9_6 http://doi.org/10.1007/978-3-642-44907-9_6 . .

T Vojíř , , , J Noskova , , , J Matas . . Robust scale-adaptive mean-shift for tracking . . Patt Recogn Lett , , 2014 . . 49 250 - - 258 . . DOI: 10.1016/j.patrec.2014.03.025 http://doi.org/10.1016/j.patrec.2014.03.025 . .

Y Wu , , , J Lim , , , MH Yang . . Online object tracking: a benchmark . . IEEE Conf on Computer Vision and Pattern Recognition , , 2013 . . p.2411 - - 2418 . . DOI: 10.1109/CVPR.2013.312 http://doi.org/10.1109/CVPR.2013.312 . .

Y Wu , , , J Lim , , , MH Yang . . Object tracking benchmark . . IEEE Trans Patt Anal Mach Intell , , 2015 . . 37 ( ( 9 ): ): 1834 - - 1848 . . DOI: 10.1109/TPAMI.2014.2388226 http://doi.org/10.1109/TPAMI.2014.2388226 . .

LX Yang , , , RS Liu , , , D Zhang , , , 等 . . Deep locationspecific tracking . . Proc 25 th ACM Int Conf on Multimedia , , 2017 . . p.1309 - - 1317 . . DOI: 10.1145/3123266.3123381 http://doi.org/10.1145/3123266.3123381 . .

A Yilmaz , , , O Javed , , , M Shah . . Object tracking: a survey . . ACM Comput Surv , , 2006 . . 38 ( ( 4 ): ): 13 DOI: 10.1145/1177352.1177355 http://doi.org/10.1145/1177352.1177355 . .

DP Young , , , JM Ferryman . . PETS metrics: on-line performance evaluation service . . IEEE Int Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance , , 2005 . . p.317 - - 324 . . DOI: 10.1109/VSPETS.2005.1570931 http://doi.org/10.1109/VSPETS.2005.1570931 . .

JC Zhang , , , YX Peng . . Object-aware aggregation with bidirectional temporal graph for video captioning . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2019 . . p.8327 - - 8336 . . . .

JM Zhang , , , SG Ma , , , S Sclaroff . . MEEM: robust tracking via multiple experts using entropy minimization . . 13 th European Conf on Computer Vision , , 2014 . . p.188 - - 203 . . DOI: 10.1007/978-3-319-10599-4_13 http://doi.org/10.1007/978-3-319-10599-4_13 . .

RF Zhang , , , T Deng , , , GH Wang , , , 等 . . A robust object tracking framework based on a reliable point assignment algorithm . . Front Inform Technol Electron Eng , , 2017 . . 18 ( ( 4 ): ): 545 - - 558 . . DOI: 10.1631/FITEE.1601464 http://doi.org/10.1631/FITEE.1601464 . .

TZ Zhang , , , S Liu , , , CS Xu , , , 等 . . Correlation particle filter for visual tracking . . IEEE Trans Image Process , , 2018 . . 27 ( ( 6 ): ): 2676 - - 2687 . . DOI: 10.1109/TIP.2017.2781304 http://doi.org/10.1109/TIP.2017.2781304 . .

WM Zuo , , , XH Wu , , , L Lin , , , 等 . . Learning support correlation filters for visual tracking . . IEEE Trans Patt Anal Mach Intell , , 2019 . . 41 ( ( 5 ): ): 1158 - - 1172 . . DOI: 10.1109/TPAMI.2018.2829180 http://doi.org/10.1109/TPAMI.2018.2829180 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Light field imaging for computer vision: a survey

Auxiliary diagnostic system for ADHD in children based on AI technology

Exploiting a depth context model in visual tracking with correlation filter

Performance analysis of visual markers for indoor navigation systems#

Inertial measurement unit-camera calibration based on incomplete inertial sensor information

Related Author

Shengyong CHEN

Meng ZHAO

Fan SHI

Chen JIA

Ming KONG

Qiang ZHU

Rong LI

Chunmao WANG

Related Institution

Engineering Research Center of Learning-Based Intelligent System (Ministry of Education), Tianjin University of Technology

Key Laboratory of Computer Vision and System (Ministry of Education), Tianjin University of Technology

Department of Psychology, The Children's Hospital, Zhejiang University School of Medicine, National Clinical Research Center for Child Health

College of Computer Science and Technology, Zhejiang University

Hikvision Research Institute

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰