Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline

Yuan HUANG; Feipeng DA

doi:10.1631/FITEE.2000508

Your Location：

Home >

Browse articles >

Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline

Regular Papers | Updated：2022-05-30

- Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline
  Enhanced Publication
- 基于双目立体匹配和B样条的三维人脸点云孔洞修补算法
- Frontiers of Information Technology & Electronic Engineering Vol. 23, Issue 3, Pages: 398-408(2022)
- Affiliations：
  
  1.School of Automation, Southeast University, Nanjing210096, China
  2.Shenzhen Research Institute of Southeast University, Shenzhen518000, China
- Author bio：
  
  ‡Corresponding authors
- Funds：
- DOI：10.1631/FITEE.2000508
  CLC： TP391.4
- Received：28 September 2020，
  
  Revised：2021-04-15，
  
  Published：2022-03
- Accepted：
Scan QR Code
Yuan HUANG, Feipeng DA. Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(3): 398-408.
DOI：

Yuan HUANG, Feipeng DA. Three-dimensional face point cloud hole-filling algorithm based on binocular stereo matching and a B-spline[J]. Frontiers of Information Technology & Electronic Engineering, 2022, 23(3): 398-408. DOI： 10.1631/FITEE.2000508.

摘要

在基于结构光的三维人脸点云数据采集过程中，由于环境、遮挡以及光照强度等因素影响，采集到的数据往往会出现孔洞区域，从而影响后续识别效果。本文提出一种采用立体匹配技术结合B样条的孔洞修补方法。算法首先利用光栅投影过程中获取的相位信息定位点云中的孔洞区域，同时提取边界点集。然后将立体匹配算法获取的人脸点云数据同光栅投影法采集的数据进行配准，在孔洞处选取初始修补控制点。再利用B样条曲线形状可由少数关键点大致描述这一特性，将控制点作为关键点放入孔洞区域进行曲面重建迭代计算。仿真使用光滑陶瓷杯和人脸模型进行，结果表明，该算法能够准确再现被测物体表面的细节和复杂形状。同时也说明所提方法具有强鲁棒性，能够在完全无先验信息的情况下对物体复杂区域实现孔洞修补，并且修补后的点云更加接近原始数据。该方法可广泛应用于需要精确人脸识别的领域。

Abstract

When obtaining three-dimensional (3D) face point cloud data based on structured light

factors related to the environment

occlusion

and illumination intensity lead to holes in the collected data

which affect subsequent recognition. In this study

we propose a hole-filling method based on stereo-matching technology combined with a B-spline. The algorithm uses phase information acquired during raster projection to locate holes in the point cloud

simultaneously extracting boundary point cloud sets. By registering the face point cloud data using the stereo-matching algorithm and the data collected using the raster projection method

some supplementary information points can be obtained at the holes. The shape of the B-spline curve can then be roughly described by a few key points

and the control points are put into the hole area as key points for iterative calculation of surface reconstruction. Simulations using smooth ceramic cups and human face models showed that our model can accurately reproduce details and accurately restore complex shapes on the test surfaces. Simulation results indicated the robustness of the method

which is able to fill holes on complex areas such as the inner side of the nose without a prior model. This approach also effectively supplements the hole information

and the patched point cloud is closer to the original data. This method could be used across a wide range of applications requiring accurate facial recognition.

关键词

Keywords

references

Bendels GH , Schnabel R , Klein R , 2006 . Detecting holes in point set surfaces . J WSCG , 14 : 89 - 96 .

Jr. Black JA , Gargesha M , Kahol K , et al. , 2016 . Framework for performance evaluation of face recognition algorithms . SPIE 4862 : 163 - 174 . https://doi.org/10.1117/12.473032 https://doi.org/10.1117/12.473032

Buchin K , van Kreveld M , Meijer H , et al. , 2009 . On planar supports for hypergraphs . Proc 17 th Int Conf on Graph Drawing , p. 345 - 356 . https://doi.org/10.1007/978-3-642-11805-0_33 https://doi.org/10.1007/978-3-642-11805-0_33

Carr JC , Beatson RK , Cherrie JB , et al. , 2001 . Reconstruction and representation of 3D objects with radial basis functions . Proc 28 th Annual Conf on Computer Graphics and Interactive Techniques , p. 67 - 76 . https://doi.org/10.1145/383259.383266 https://doi.org/10.1145/383259.383266

Chen H , Ma SW , Nuechter A , 2016 . Non-synchronous point cloud algorithm for 3D reconstruction based on laser scanning and SFM . Chin J Sci Instrum , 37 ( 5 ): 1148 - 1157 (in Chinese) . https://doi.org/10.3969/j.issn.0254-3087.2016.05.024 https://doi.org/10.3969/j.issn.0254-3087.2016.05.024

Chui CK , Lai MJ , 2000 . Filling polygonal holes using C 1 cubic triangular spline patches . Comput Aided Geom Des , 17 ( 4 ): 297 - 307 . https://doi.org/10.1016/S0167-8396(00)00005-4 https://doi.org/10.1016/S0167-8396(00)00005-4

Floater MS , Reimers M , 2001 . Meshless parameterization and surface reconstruction . Comput Aided Geom Des , 18 ( 2 ): 77 - 92 . https://doi.org/10.1016/S0167-8396(01)00013-9 https://doi.org/10.1016/S0167-8396(01)00013-9

Furukawa Y , Ponce J , 2010 . Accurate, dense, and robust multiview stereopsis . IEEE Trans Patt Anal Mach Intell , 32 ( 8 ): 1362 - 1376 . https://doi.org/10.1109/TPAMI.2009.161 https://doi.org/10.1109/TPAMI.2009.161

Gilani ZS , Mian A , 2016 . Towards large-scale 3D face recognition . Proc Int Conf on Digital Image Computing : Techniques and Applications , p. 1 - 8 . https://doi.org/10.1109/DICTA.2016.7797090 https://doi.org/10.1109/DICTA.2016.7797090

Goesele M , Curless B , Seitz SM , 2006 . Multi-view stereo revisited . Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition , p. 2402 - 2409 . https://doi.org/10.1109/CVPR.2006.199 https://doi.org/10.1109/CVPR.2006.199

Huang XS , Zhang J , Fan LX , et al. , 2017 . A systematic approach for cross-source point cloud registration by preserving macro and micro structures . IEEE Trans Image Process , 26 ( 7 ): 3261 - 3276 . https://doi.org/10.1109/TIP.2017.2695888 https://doi.org/10.1109/TIP.2017.2695888

Huang Y , Da FP , 2019 . Registration algorithm for point cloud based on normalized cross-correlation . IEEE Access , 7 : 137136 - 137146 . https://doi.org/10.1109/ACCESS.2019.2942127 https://doi.org/10.1109/ACCESS.2019.2942127

Huang Y , Da FP , Tao HJ , 2015 . An automatic registration algorithm for point cloud based on feature extraction . Chin J Lasers , 42 ( 3 ): 308002 (in Chinese) . https://doi.org/10.3788/CJL201542.0308002 https://doi.org/10.3788/CJL201542.0308002

Jeong Y , Bok Y , Kim JS , et al. , 2011 . Complementation of cameras and lasers for accurate 6D SLAM: from correspondences to bundle adjustment . Proc IEEE Int Conf on Robotics and Automation , p. 3581 - 3588 . https://doi.org/10.1109/ICRA.2011.5979568 https://doi.org/10.1109/ICRA.2011.5979568

Jun Y , 2005 . A piecewise hole filling algorithm in reverse engineering . Comput-Aided Des , 37 ( 2 ): 263 - 270 . https://doi.org/10.1016/j.cad.2004.06.012 https://doi.org/10.1016/j.cad.2004.06.012

Kurlin V , 2014 . A fast and robust algorithm to count topologically persistent holes in noisy clouds . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 1458 - 1463 . https://doi.org/10.1109/CVPR.2014.189 https://doi.org/10.1109/CVPR.2014.189

Liu K , Zhou CH , Wei SB , et al. , 2014 . Optimized stereo matching in binocular three-dimensional measurement system using structured light . Appl Opt , 53 ( 26 ): 6083 - 6090 . https://doi.org/10.1364/AO.53.006083 https://doi.org/10.1364/AO.53.006083

Liu YJ , Wang M , Zhang HL , et al. , 2016 . Strategy of classification and repairing for hole of incomplete point clouds based on fuzzy inference . J Comput Theor Nanosci , 13 ( 11 ): 8227 - 8233 . https://doi.org/10.1166/jctn.2016.5961 https://doi.org/10.1166/jctn.2016.5961

Nguyen VS , Trinh TH , Tran MH , 2015 . Hole boundary detection of a surface of 3D point clouds . Proc Int Conf on Advanced Computing and Applications , p. 124 - 129 . https://doi.org/10.1109/ACOMP.2015.12 https://doi.org/10.1109/ACOMP.2015.12

Orriols X , Binefa X , 2003 . Finding breaking curves in 3D surface . Proc 1 st Iberian Conf on Pattern Recognition and Image Analysis , p. 681 - 688 . https://doi.org/10.1007/978-3-540-44871-6_79 https://doi.org/10.1007/978-3-540-44871-6_79

O’Toole AJ , An XB , Dunlop J , et al. , 2012 . Comparing face recognition algorithms to humans on challenging tasks . ACM Trans Appl Percept , 9 ( 4 ): 16 . https://doi.org/10.1145/2355598.2355599 https://doi.org/10.1145/2355598.2355599

Pan YH , 2019 . On visual knowledge . Front Inform Technol Electron Eng , 20 ( 8 ): 1021 - 1025 . https://doi.org/10.1631/FITEE.1910001 https://doi.org/10.1631/FITEE.1910001

Pan YH , 2021 . Miniaturized five fundamental issues about visual knowledge . Front Inform Technol Electron Eng , 22 ( 5 ): 615 - 618 . https://doi.org/10.1631/FITEE.2040000 https://doi.org/10.1631/FITEE.2040000

Panchetti M , Pernot JP , Véron P , 2010 . Towards recovery of complex shapes in meshes using digital images for reverse engineering applications . Comput-Aided Des , 42 ( 8 ): 693 - 707 . https://doi.org/10.1016/j.cad.2010.01.004 https://doi.org/10.1016/j.cad.2010.01.004

Pernot JP , Moraru G , Véron P , 2007 . Repairing triangle meshes built from scanned point cloud . J Eng Des , 18 ( 5 ): 459 - 473 . https://doi.org/10.1080/09544820701403797 https://doi.org/10.1080/09544820701403797

Quinsat Y , Lartigue C , 2015 . Filling holes in digitized point cloud using a morphing-based approach to preserve volume characteristics . Int J Adv Manuf Technol , 81 ( 1-4 ): 411 - 421 . https://doi.org/10.1007/s00170-015-7185-0 https://doi.org/10.1007/s00170-015-7185-0

Russell , Stuart J , Norvig P , 2010 . Artificial intelligence: a modern approach . Appl Mech Mater , 263 ( 5 ): 2829 - 2833 .

Schaffer M , Grosse M , Harendt B , et al. , 2011 . High-speed three-dimensional shape measurements of objects with laser speckles and acousto-optical deflection . Opt Lett , 36 ( 16 ): 3097 - 3099 . https://doi.org/10.1364/OL.36.003097 https://doi.org/10.1364/OL.36.003097

Shi LM , Guo FS , Hu ZY , 2011 . An improved PMVS through scene geometric information . Acta Autom Sin , 37 ( 5 ): 560 - 568 (in Chinese) . https://doi.org/10.3724/SP.J.1004.2011.00560 https://doi.org/10.3724/SP.J.1004.2011.00560

Stone EE , Skubic M , 2015 . Fall detection in homes of older adults using the Microsoft Kinect . IEEE J Biomed Health Inform , 19 ( 1 ): 290 - 301 . https://doi.org/10.1109/JBHI.2014.2312180 https://doi.org/10.1109/JBHI.2014.2312180

Wang JN , Oliveira MM , 2007 . Filling holes on locally smooth surfaces reconstructed from point clouds . Image Vis Comput , 25 ( 1 ): 103 - 113 . https://doi.org/10.1016/j.imavis.2005.12.006 https://doi.org/10.1016/j.imavis.2005.12.006

Zheng EL , Wu CC , 2015 . Structure from motion using structure-less resection . Proc IEEE Int Conf on Computer Vision , p. 2075 - 2083 . https://doi.org/10.1109/ICCV.2015.240 https://doi.org/10.1109/ICCV.2015.240

Views

189

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰