On 3D face reconstruction via cascaded regression in shape space

LIU Feng; ZENG Dan; LI Jing; ZHAO Qi-jun

doi:10.1631/FITEE.1700253

Your Location：

Home >

Browse articles >

On 3D face reconstruction via cascaded regression in shape space

Regular Papers | Updated：2022-06-06

- On 3D face reconstruction via cascaded regression in shape space
- 形状空间下基于级联回归的三维人脸重建
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 12, Pages: 1978-1990(2017)
- Affiliations：
  
  College of Computer Science, Sichuan University, Chengdu 610065, China
- Author bio：
  
  Qi-jun ZHAO, E-mail: qjzhao@scu.edu.cn
- Funds：
- DOI：10.1631/FITEE.1700253
  CLC： TP319
- Received：16 April 2017，
  
  Revised：2017-12-20，
  
  Published：2017-12
- Accepted：
Scan QR Code
LIU Feng, ZENG Dan, LI Jing, et al. On 3D face reconstruction via cascaded regression in shape space[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(12): 1978-1990.
DOI：

LIU Feng, ZENG Dan, LI Jing, et al. On 3D face reconstruction via cascaded regression in shape space[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(12): 1978-1990. DOI： 10.1631/FITEE.1700253.

摘要

最近，级联回归方法在基于二维图像的三维人脸重建中取得了良好的效果。本文主要探究了基于级联回归三维人脸重建中四个未解决的重要问题：（1）二维特征点数量的影响；（2）三维点云数量的影响；（3）自动二维特征点的检测方法；（4）算法的收敛。本文专门设计了一个简单有效的基于形状空间的级联回归三维重建算法。该算法有效利用各类独立的自动二维特征点检测算法，且重建的三维人脸具有和输入图像相同的姿态和表情。同时在训练中，本文设计了一种随机扰动检二维特征点位置信息的策略，使得算法更有效且更鲁棒。最后，通过大量的对比实验证明了该算法不仅有着良好的精度和计算效率，而且对加深级联回归三维重建方法的理解有着启发意义。

Abstract

Cascaded regression has been recently applied to reconstruct 3D faces from single 2D images directly in shape space

and has achieved state-of-the-art performance. We investigate thoroughly such cascaded regression based 3D face reconstruction approaches from four perspectives that are not well been studied: (1) the impact of the number of 2D landmarks; (2) the impact of the number of 3D vertices; (3) the way of using standalone automated landmark detection methods; (4) the convergence property. To answer these questions

a simplified cascaded regression based 3D face reconstruction method is devised. This can be integrated with standalone automated landmark detection methods and reconstruct 3D face shapes that have the same pose and expression as the input face images

rather than normalized pose and expression. An effective training method is also proposed by disturbing the automatically detected landmarks. Comprehensive evaluation experiments have been carried out to compare to other 3D face reconstruction methods. The results not only deepen the understanding of cascaded regression based 3D face reconstruction approaches

but also prove the effectiveness of the proposed method.

关键词

Keywords

references

R. Abiantun , U. Prabhu , M. Savvides . Sparse feature extraction for pose-tolerant face recognition . IEEE Trans. Patt. Anal. Mach. Intell. , 2014 . 36 ( 10 ): 2061 - 2073 . DOI: 10.1109/TPAMI.2014.2313124 http://doi.org/10.1109/TPAMI.2014.2313124 .

O. Aldrian , W.A.P. Smith . Inverse rendering of faces with a 3D morphable model . IEEE Trans. Patt. Anal. Mach. Intell. , 2013 . 35 ( 5 ): 1080 - 1093 . DOI: 10.1109/TPAMI.2012.206 http://doi.org/10.1109/TPAMI.2012.206 .

A.D. Bagdanov , A. Del Bimbo , I. Masi . The florence 2D/3D hybrid face dataset . Workshop on Human Gesture and Behavior Understanding , 2011 . p.79 - 80 . DOI: 10.1145/2072572.2072597 http://doi.org/10.1145/2072572.2072597 .

J.T. Barron , J. Malik . Shape, albedo, and illumination from a single image of an unknown object . IEEE Conf. on Computer Vision and Pattern Recognition , 2012 . p.334 - 341 . DOI: 10.1109/CVPR.2012.6247693 http://doi.org/10.1109/CVPR.2012.6247693 .

A. Bas , W.A.P. Smith , T. Bolkart , 等 . Fitting a 3D morphable model to edges: a comparison between hard and soft correspondences . IEEE Asian Conf. on Computer Vision , 2016 . p.377 - 391 . DOI: 10.1007/978-3-319-54427-4_28 http://doi.org/10.1007/978-3-319-54427-4_28 .

V. Blanz , T. Vetter . A morphable model for the synthesis of 3D faces . Proc. SIGGRAPH , 1999 . p.187 - 194 . DOI: 10.1145/311535.311556 http://doi.org/10.1145/311535.311556 .

V. Blanz , T. Vetter . Face recognition based on fitting a 3D morphable model . IEEE Trans. Patt. Anal. Mach. Intell. , 2003 . 25 ( 9 ): 1063 - 1074 . DOI: 10.1109/TPAMI.2003.1227983 http://doi.org/10.1109/TPAMI.2003.1227983 .

J. Booth , A. Roussos , S. Zafeiriou , 等 . A 3D morphable model learnt from 10 000 faces . IEEE Conf. on Computer Vision and Pattern Recognition , 2016 . p.5543 - 5552 . DOI: 10.1109/CVPR.2016.598 http://doi.org/10.1109/CVPR.2016.598 .

C. Cao , Q.M. Hou , K. Zhou . Displaced dynamic expression regression for real-time facial tracking and animation . ACM Trans. Graph. , 2014a . 33 ( 4 ): 43.1 - 43.10 . DOI: 10.1145/2601097.2601204 http://doi.org/10.1145/2601097.2601204 .

C. Cao , Y.L. Weng , S. Zhou , 等 . Facewarehouse: a 3D facial expression database for visual computing . IEEE Trans. Vis. Comput. Graph. , 2014b . 20 ( 3 ): 413 - 425 . DOI: 10.1109/TVCG.2013.249 http://doi.org/10.1109/TVCG.2013.249 .

C. Cao , H.Z. Wu , Y.L. Weng , 等 . Real-time facial animation with image-based dynamic avatars . ACM Trans. Graph. , 2016 . 35 ( 4 ): 126.1 - 126.12 . DOI: 10.1145/2897824.2925873 http://doi.org/10.1145/2897824.2925873 .

B. Chu , S. Romdhani , L.M. Chen . 3D-aided face recognition robust to expression and pose variations . IEEE Conf. on Computer Vision and Pattern Recognition , 2014 . p.1907 - 1914 . DOI: 10.1109/CVPR.2014.245 http://doi.org/10.1109/CVPR.2014.245 .

H. Han , A.K. Jain . 3D face texture modeling from uncalibrated frontal and profile images . Int. Conf. on Biometrics: Theory, Applications and Systems , 2012 . p.223 - 230 . DOI: 10.1109/BTAS.2012.6374581 http://doi.org/10.1109/BTAS.2012.6374581 .

B.K.P. Horn , M.J. Brooks . Shape from Shading , : Cambridge, MA, USA MIT Press , 1989 .

J.L. Hu , J.W. Lu , Y.P. Tan . Discriminative deep metric learning for face verification in the wild . IEEE Conf. on Computer Vision and Pattern Recognition , 2014 . p.1875 - 1882 . DOI: 10.1109/CVPR.2014.242 http://doi.org/10.1109/CVPR.2014.242 .

A. Jourabloo , X.M. Liu . Pose-invariant 3D face alignment . IEEE Int. Conf. on Computer Vision , 2015 . p.3694 - 3702 . DOI: 10.1109/ICCV.2015.421 http://doi.org/10.1109/ICCV.2015.421 .

A. Jourabloo , X.M. Liu . Large-pose face alignment via CNN-based dense 3D model fitting . IEEE Conf. on Computer Vision and Pattern Recognition , 2016 . p.4188 - 4196 . DOI: 10.1109/CVPR.2016.454 http://doi.org/10.1109/CVPR.2016.454 .

A. Jourabloo , X.M. Liu . Pose-invariant face alignment via CNN-based dense 3D model fitting . Int. J. Comput. Vis. , 2017 . 4 1 - 17 . DOI: 10.1007/s11263-017-1012-z http://doi.org/10.1007/s11263-017-1012-z .

V. Kazemi , J. Sullivan . One millisecond face alignment with an ensemble of regression trees . IEEE Conf. on Computer Vision and Pattern Recognition , 2014 . p.1867 - 1874 . DOI: 10.1109/CVPR.2014.241 http://doi.org/10.1109/CVPR.2014.241 .

I. Kemelmacher-Shlizerman , R. Basri . 3D face reconstruction from a single image using a single reference face shape . IEEE Trans. Patt. Anal. Mach. Intell. , 2011 . 33 ( 2 ): 394 - 405 . DOI: 10.1109/TPAMI.2010.63 http://doi.org/10.1109/TPAMI.2010.63 .

M. Köstinger , P. Wohlhart , P.M. Roth , 等 . Annotated facial landmarks in the wild: a large-scale, realworld database for facial landmark localization . IEEE Int. Conf. on Computer Vision Workshops , 2011 . p.2144 - 2151 . DOI: 10.1109/ICCVW.2011.6130513 http://doi.org/10.1109/ICCVW.2011.6130513 .

J. Li , S.Q. Long , D. Zeng , 等 . Example-based 3D face reconstruction from uncalibrated frontal and profile images . IEEE Int. Conf. on Biometrics , 2015 . p.193 - 200 . DOI: 10.1109/ICB.2015.7139051 http://doi.org/10.1109/ICB.2015.7139051 .

X. Li , Y.D. Xu , Q. Lv , 等 . Affine-transformation parameters regression for face alignment . IEEE Signal Process. Lett. , 2016 . 23 ( 1 ): 55 - 59 . DOI: 10.1109/LSP.2015.2499778 http://doi.org/10.1109/LSP.2015.2499778 .

F. Liu , D. Zeng , Q.J. Zhao , 等 . Joint face alignment and 3D face reconstruction . European Conf. on Computer Vision , 2016 . p.545 - 560 . DOI: 10.1007/978-3-319-46454-1_33 http://doi.org/10.1007/978-3-319-46454-1_33 .

P. Paysan , R. Knothe , B. Amberg , 等 . A 3D face model for pose and illumination invariant face recognition . IEEE Conf. on Advanced Video and Signal-based Surveillance , 2009 . p.296 - 301 . DOI: 10.1109/AVSS.2009.58 http://doi.org/10.1109/AVSS.2009.58 .

J.F. Ren , X.D. Jiang , J.S. Yuan . Face and facial expressions recognition and analysis . Context Aware Human-Robot and Human-Agent Interaction , 2016 . p.3 - 29 . DOI: 10.1007/978-3-319-19947-4_1 http://doi.org/10.1007/978-3-319-19947-4_1 .

S. Romdhani , T. Vetter . Estimating 3D shape and texture using pixel intensity, edges, specular highlights, texture constraints and a prior . IEEE Conf. on Computer Vision and Pattern Recognition , 2005 . p.986 - 993 . DOI: 10.1109/CVPR.2005.145 http://doi.org/10.1109/CVPR.2005.145 .

C. Sagonas , G. Tzimiropoulos , S. Zafeiriou , 等 . 300 faces in-the-wild challenge: the first facial landmark localization challenge . IEEE Int. Conf. on Computer Vision Workshop , 2013 . p.397 - 403 . DOI: 10.1109/ICCVW.2013.59 http://doi.org/10.1109/ICCVW.2013.59 .

Y. Sun , X.G. Wang , X.O. Tang . Deep convolutional network cascade for facial point detection . IEEE Conf. on Computer Vision and Pattern Recognition , 2013 . p.3476 - 3483 . DOI: 10.1109/CVPR.2013.446 http://doi.org/10.1109/CVPR.2013.446 .

S. Suwajanakorn , I. Kemelmacher-Shlizerman , S.M. Seitz . Total moving face reconstruction . European Conf. on Computer Vision , 2014 . p.796 - 812 . DOI: 10.1007/978-3-319-10593-2_52 http://doi.org/10.1007/978-3-319-10593-2_52 .

X.H. Xiong , F. de la Torre . Supervised descent method and its applications to face alignment . IEEE Conf. on Computer Vision and Pattern Recognition , 2013 . p.532 - 539 . DOI: 10.1109/CVPR.2013.75 http://doi.org/10.1109/CVPR.2013.75 .

L.J. Yin , X.Z. Wei , Y. Sun , 等 . A 3D facial expression database for facial behavior research . IEEE Int. Conf. on Automatic Face and Gesture Recognition , 2006 . p.211 - 216 . DOI: 10.1109/FGR.2006.6 http://doi.org/10.1109/FGR.2006.6 .

D. Zeng , Q.J. Zhao , S.Q. Long , 等 . Examplar coherent 3D face reconstruction from forensic mugshot database . Image Vis. Comput. , 2017 . 58 193 - 203 . DOI: 10.1016/j.imavis.2016.03.001 http://doi.org/10.1016/j.imavis.2016.03.001 .

Z.P. Zhang , P. Luo , C.L. Chen , 等 . Facial landmark detection by deep multi-task learning . European Conf. on Computer Vision , 2014 . p.94 - 108 . DOI: 10.1007/978-3-319-10599-4_7 http://doi.org/10.1007/978-3-319-10599-4_7 .

X.W. Zhou , S. Leonardos , X.Y. Hu , 等 . 3D shape estimation from 2D landmarks: a convex relaxation approach . IEEE Conf. on Computer Vision and Pattern Recognition , 2015 . p.4447 - 4455 . DOI: 10.1109/CVPR.2015.7299074 http://doi.org/10.1109/CVPR.2015.7299074 .

S.Z. Zhu , C. Li , C.L. Chen , 等 . Face alignment by coarse-to-fine shape searching . IEEE Conf. on Computer Vision and Pattern Recognition , 2015 . p.4998 - 5006 . DOI: 10.1109/CVPR.2015.7299134 http://doi.org/10.1109/CVPR.2015.7299134 .

X.X. Zhu , D. Ramanan . Face detection, pose estimation, and landmark localization in the wild . IEEE Conf. on Computer Vision and Pattern Recognition , 2012 . p.2879 - 2886 . DOI: 10.1109/CVPR.2012.6248014 http://doi.org/10.1109/CVPR.2012.6248014 .

X.Y. Zhu , D. Yi , Z. Lei , 等 . Robust 3D morphable model fitting by sparse SIFT flow . IEEE Int. Conf. on Pattern Recognition , 2014 . p.4044 - 4049 . DOI: 10.1109/ICPR.2014.693 http://doi.org/10.1109/ICPR.2014.693 .

X.Y. Zhu , Z. Lei , J.J. Yan , 等 . High-fidelity pose and expression normalization for face recognition in the wild . IEEE Conf. on Computer Vision and Pattern Recognition , 2015 . p.787 - 796 . DOI: 10.1109/CVPR.2015.7298679 http://doi.org/10.1109/CVPR.2015.7298679 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

GeeNet: robust and fast point cloud completion for ground elevation estimation towards autonomous vehicles

Related Author

Liwen LIU

Weidong YANG

Ben FEI

Related Institution

Shanghai Key Laboratory of Data Science, School of Computer Science, Fudan University

Zhuhai Fudan Innovation Institute

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰