Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics

Ping-ping WU; Hong LIU; Xue-wu ZHANG; Yuan GAO

doi:10.1631/FITEE.1600041

Your Location：

Home >

Browse articles >

Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics

Regular Papers | Updated：2022-05-19

- Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics
- 基于特定区域纹理描述和面部动态变化的自发性微笑判别技术
- Frontiers of Information Technology & Electronic Engineering Vol. 18, Issue 7, Pages: 955-967(2017)
- Affiliations：
  
  MOE Key Laboratory of Machine Perception, Peking University, Beijing 100871, China
  Engineering Lab on Intelligent Perception for Internet of Things, Peking University Shenzhen Graduate School, Shenzhen 518055, China
  Department of Computer Science, Christian-Albrechts University, Kiel 24118, Germany
- Author bio：
  
  [ "Ping-ping WU, E-mail:pingpingwu@pku.edu.cn" ]
  Hong LIU, E-mail:hongliu@pku.edu.cn
  [ "Xue-wu ZHANG, E-mail:zhangxuewu@sz.pku.edu.cn" ]
  [ "Yuan GAO, E-mail:yuan.gao@stu.uni-kiel.de" ]
- Funds：
- DOI：10.1631/FITEE.1600041
  CLC： TP37
- Received：27 January 2016，
  
  Revised：2017-;6-;3，
  
  Published：2017-07
- Accepted：
Scan QR Code
Ping-ping WU, Hong LIU, Xue-wu ZHANG, et al. Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(7): 955-967.
DOI：

Ping-ping WU, Hong LIU, Xue-wu ZHANG, et al. Spontaneous versus posed smile recognition via region-specific texture descriptor and geometric facial dynamics[J]. Frontiers of Information Technology & Electronic Engineering, 2017, 18(7): 955-967. DOI： 10.1631/FITEE.1600041.

摘要

微笑作为一种典型的生物多样性特征信号，在社会交往中有较大影响力，它揭示了人的情感感受和内心状态。自发性的微笑与假笑由不同大脑系统发出，在形态学和动力学上均存在差异。区分这两种类型的微笑仍具有挑战性，因为其中细微差别很难被肉眼观察到，仍有待被识别捕捉。已有相关研究大多是提取自发性微笑的几何特征，而外观特征并没有被充分利用，导致纹理信息的丢失。本文提出一种基于特定区域纹理描述来表示不同面部区域的局部模式变化，从而弥补几何特征研究的局限性。每个面部区域的时间相位是通过计算相应的面部区域强度来划分，而非仅考虑嘴巴区域强度。同时利用支持向量机的中层融合策略，将两种特征类型结合起来。实验结果表明，本文提出的外观表示法及其与基于几何形状的人脸动力学的结合技术，在BBC、SPOS、MMI和UvA-NEMO四个基准数据库中得到很好的应用。

Abstract

As a typical biometric cue with great diversities

smile is a fairly influential signal in social interaction

which reveals the emotional feeling and inner state of a person. Spontaneous and posed smiles initiated by different brain systems have differences in both morphology and dynamics. Distinguishing the two types of smiles remains challenging as discriminative subtle changes need to be captured

which are also uneasily observed by human eyes. Most previous related works about spontaneous versus posed smile recognition concentrate on extracting geometric features while appearance features are not fully used

leading to the loss of texture information. In this paper

we propose a region-specific texture descriptor to represent local pattern changes of different facial regions and compensate for limitations of geometric features. The temporal phase of each facial region is divided by calculating the intensity of the corresponding facial region rather than the intensity of only the mouth region. A mid-level fusion strategy of support vector machine is employed to combine the two feature types. Experimental results show that both our proposed appearance representation and its combination with geometry-based facial dynamics achieve favorable performances on four baseline databases: BBC

SPOS

MMI

and UvA-NEMO.

关键词

Keywords

references

Z. Ambadar , , , J.F. Cohn , , , L. Ian Reed . . All smiles are not created equal: morphology and timing of smiles perceived as amused, polite, and embarrassed/nervous . . J. Nonverb. Behav. , , 2009 . . 33 ( ( 1 ): ): 17 - - 34 . . DOI: 10.1007/s10919-008-0059-5 http://doi.org/10.1007/s10919-008-0059-5 . .

S. Baron-Cohen , , , H.A. Ring , , , E.T. Bullmore , , , 等 . . The amygdala theory of autism . . Neurosci. Biobehav. Rev. , , 2000 . . 24 ( ( 3 ): ): 355 - - 364 . . DOI: 10.1016/S0149-7634(00)00011-7 http://doi.org/10.1016/S0149-7634(00)00011-7 . .

X.P. Burgos-Artizzu , , , P. Perona , , , P. Dollár . . Robust face landmark estimation under occlusion . . Proc. IEEE Int. Conf. on Computer Vision , , 2013 . . p.1513 - - 1520 . . DOI: 10.1109/ICCV.2013.191 http://doi.org/10.1109/ICCV.2013.191 . .

M.G. Calvo , , , L. Nummenmaa . . Time course of discrimination between emotional facial expressions: the role of visual saliency . . Vis. Res. , , 2011 . . 51 ( ( 15 ): ): 1751 - - 1759 . . DOI: 10.1016/j.visres.2011.06.001 http://doi.org/10.1016/j.visres.2011.06.001 . .

M.G. Calvo , , , A. Gutiérrez-García , , , P. Avero , , , 等 . . Attentional mechanisms in judging genuine and fake smiles: eye-movement patterns . . Emotion , , 2013 . . 13 ( ( 4 ): ): 792 - - 802 . . DOI: 10.1037/a0032317 http://doi.org/10.1037/a0032317 . .

X. Cao , , , Y. Wei , , , F. Wen , , , 等 . . Face alignment by explicit shape regression . . Int. J. Comput. Vis. , , 2014 . . 107 ( ( 2 ): ): 177 - - 190 . . DOI: 10.1007/s11263-013-0667-3 http://doi.org/10.1007/s11263-013-0667-3 . .

J.F. Cohn , , , K.L. Schmidt . . The timing of facial motion in posed and spontaneous smiles . . Int. J. Wavel. Multiresol. Inform. Process. , , 2004 . . 2 ( ( 2 ): ): 121 - - 132 . . DOI: 10.1142/S021969130400041X http://doi.org/10.1142/S021969130400041X . .

T.F. Cootes , , , G.J. Edwards , , , C.J. Taylor . . Active appearance models . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2001 . . 23 ( ( 6 ): ): 681 - - 685 . . DOI: 10.1109/34.927467 http://doi.org/10.1109/34.927467 . .

N. Dalal , , , B. Triggs . . Histograms of oriented gradients for human detection . . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , , 2005 . . p.886 - - 893 . . DOI: 10.1109/CVPR.2005.177 http://doi.org/10.1109/CVPR.2005.177 . .

H. Dibeklioğlu , , , R. Valenti , , , A.A. Salah , , , 等 . . Eyes do not lie: spontaneous versus posed smiles . . Proc. Int. Conf. on Multimedia , , 2010 . . p.703 - - 706 . . DOI: 10.1145/1873951.1874056 http://doi.org/10.1145/1873951.1874056 . .

H. Dibeklioğlu , , , A. Salah , , , T. Gevers . . Are you really smiling at me? . . Spontaneous versus posed enjoymentsmiles. Proc. European Conf. on Computer Vision , , 2012 . . p.525 - - 538 . . DOI: 10.1007/978-3-642-33712-3_38 http://doi.org/10.1007/978-3-642-33712-3_38 . .

H. Dibeklioğlu , , , A. Salah , , , T. Gevers . . Recognition of genuine smiles . . IEEE Trans. Multimed. , , 2015 . . 17 ( ( 3 ): ): 279 - - 294 . . DOI: 10.1109/TMM.2015.2394777 http://doi.org/10.1109/TMM.2015.2394777 . .

P. Dollár , , , P. Welinder , , , P. Perona . . Cascaded pose regression . . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , , 2010 . . p.1078 - - 1085 . . DOI: 10.1109/CVPR.2010.5540094 http://doi.org/10.1109/CVPR.2010.5540094 . .

P. Ekman . . Telling Lies: Clues to Deceit in the Marketplace, Politics, and Marriage . . W. W. Norton & Company, New York , , 2009 . . p.140 - - 143 . . . .

P. Ekman , , , E.L. Rosenberg . . What the Face Reveals: Basic and Applied Studies of Spontaneous Expression Using the Facial Action Coding System (FACS) , , Oxford University Press , , 1997 . . .

M.G. Frank , , , P. Ekman . . Not all smiles are created equal: the differences between enjoyment and nonenjoyment smiles . . Humor , , 1993 . . 6 ( ( 1 ): ): 9 - - 26 . . DOI: 10.1515/humr.1993.6.1.9 http://doi.org/10.1515/humr.1993.6.1.9 . .

M. Hoque , , , D. McDuff , , , R. Picard . . Exploring temporal patterns in classifying frustrated and delighted smiles . . IEEE Trans. Affect. Comput. , , 2012 . . 3 ( ( 3 ): ): 323 - - 334 . . DOI: 10.1109/T-AFFC.2012.11 http://doi.org/10.1109/T-AFFC.2012.11 . .

M.R. Khokher , , , A. Bouzerdoum , , , S.L. Phung . . Crowd behavior recognition using dense trajectories . . Proc. Int. Conf. on Digital lmage Computing: Techniques and Applications , , 2014 . . p.1 - - 7 . . DOI: 10.1109/DICTA.2014.7008098 http://doi.org/10.1109/DICTA.2014.7008098 . .

V. Le , , , J. Brandt , , , Z. Lin , , , 等 . . Interactive facial feature localization . . Proc. European Conf. on Computer Vision , , 2012 . . p.679 - - 692 . . DOI: 10.1007/978-3-642-33712-3_49 http://doi.org/10.1007/978-3-642-33712-3_49 . .

W.S. Li , , , C.L. Zhou , , , J.T. Xu . . A novel face recognition method with feature combination . . J. Zhejiang Univ.-Sci. , , 2005 . . 6A ( ( 5 ): ): 454 - - 459 . . DOI: 10.1631/jzus.2005.A0454 http://doi.org/10.1631/jzus.2005.A0454 . .

H. Liu , , , X. Sun . . A partial least squares based ranker for fast and accurate age estimation . . Proc. IEEE Int. Conf. on Acoustics, Speech and Signal Processing , , 2016 . . p.2792 - - 2796 . . DOI: 10.1109/ICASSP.2016.7472186 http://doi.org/10.1109/ICASSP.2016.7472186 . .

H. Liu , , , P. Wu . . Comparison of methods for smile deceit detection by training AU6 and AU12 simultaneously . . Proc. IEEE Int. Conf. on Image Processing , , 2012 . . p.1805 - - 1808 . . DOI: 10.1109/ICIP.2012.6467232 http://doi.org/10.1109/ICIP.2012.6467232 . .

H. Liu , , , Y. Gao , , , C. Wang . . Gender identification in unconstrained scenarios using self-similarity of gradients features . . Proc. IEEE Int. Conf. on Image Processing , , 2014 . . p.5911 - - 5915 . . DOI: 10.1109/ICIP.2014.7026194 http://doi.org/10.1109/ICIP.2014.7026194 . .

A. Miehlke , , , U. Fisch , , , C.M. Eneroth . . Surgery of the Facial Nerve . . Saunders, Philadelphia. , , 1973 . . .

T. Ojala , , , M. Pietikäinen , , , T. Mäenpää . . Multiresolution gray-scale and rotation invariant texture classification with local binary patterns . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2002 . . 24 ( ( 7 ): ): 971 - - 987 . . DOI: 10.1109/TPAMI.2002.1017623 http://doi.org/10.1109/TPAMI.2002.1017623 . .

H. Peng , , , F. Long , , , C. Ding . . Feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2005 . . 27 ( ( 8 ): ): 1226 - - 1238 . . DOI: 10.1109/TPAMI.2005.159 http://doi.org/10.1109/TPAMI.2005.159 . .

T. Pfister , , , X. Li , , , G. Zhao , , , 等 . . Differentiating spontaneous from posed facial expressions within a generic facial expression recognition framework . . Proc. IEEE Int. Conf. on Computer Vision Workshops , , 2011a . . p.868 - - 875 . . DOI: 10.1109/ICCVW.2011.6130343 http://doi.org/10.1109/ICCVW.2011.6130343 . .

T. Pfister , , , X. Li , , , G. Zhao , , , 等 . . Recognising spontaneous facial micro-expressions . . Proc. IEEE Int. Conf. on Computer Vision , , 2011b . . p.1449 - - 1456 . . DOI: 10.1109/ICCV.2011.6126401 http://doi.org/10.1109/ICCV.2011.6126401 . .

W.E. Rinn . . The neuropsychology of facial expression: a review of the neurological and psychological mechanisms for producing facial expressions . . Psychol. Bull. , , 1984 . . 95 ( ( 1 ): ): 52 - - 77 . . DOI: 10.1037/0033-2909.95.1.52 http://doi.org/10.1037/0033-2909.95.1.52 . .

E. Sariyanidi , , , H. Gunes , , , A. Cavallaro . . Automatic analysis of facial affect: a survey of registration, representation, and recognition . . IEEE Trans. Patt. Anal. Mach. Intell. , , 2015 . . 37 ( ( 6 ): ): 1113 - - 1133 . . DOI: 10.1109/TPAMI.2014.2366127 http://doi.org/10.1109/TPAMI.2014.2366127 . .

X.B. Shen , , , Q. Wu , , , X.L. Fu . . Effects of the duration of expressions on the recognition of microexpressions . . J. Zhejiang Univ.-Sci. B , , 2012 . . 13 ( ( 3 ): ): 221 - - 230 . . DOI: 10.1631/jzus.B1100063 http://doi.org/10.1631/jzus.B1100063 . .

M. Valstar , , , M. Pantic . . Induced disgust, happiness and surprise: an addition to the MMI facial expression database . . Proc. 3rd Int. Workshop on EMOTION (satellite of LREC): Corpora for Research on Emotion and Affect , , 2010 . . p.65 - - 70 . . . .

M.F. Valstar , , , H. Gunes , , , M. Pantic . . How to distinguish posed from spontaneous smiles using geometric features . . Proc. Int. Conf. on Multimodal Interfaces , , 2007 . . p.38 - - 45 . . DOI: 10.1145/1322192.1322202 http://doi.org/10.1145/1322192.1322202 . .

J. Wang , , , J. Yang , , , K. Yu , , , 等 . . Locality-constrained linear coding for image classification . . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , , 2010 . . p.3360 - - 3367 . . DOI: 10.1109/CVPR.2010.5540018 http://doi.org/10.1109/CVPR.2010.5540018 . .

X. Wang , , , L. Wang , , , Y. Qiao . . A comparative study of encoding, pooling and normalization methods for action recognition . . Proc. Asian Conf. on Computer Vision , , 2012 . . p.572 - - 585 . . DOI: 10.1007/978-3-642-37431-9_44 http://doi.org/10.1007/978-3-642-37431-9_44 . .

J. Whitehill , , , M.S. Bartlett , , , J.R. Movellan . . Automatic facial expression recognition . . In: Gratch J., Marsella, S. (Eds.), Social Emotions in Nature Artifact. Oxford Scholarship Online. , , 2013 . . DOI: 10.1093/acprof:oso/9780195387643.003.0007 http://doi.org/10.1093/acprof:oso/9780195387643.003.0007 . .

P.P. Wu , , , H. Liu , , , X.W. Zhang . . Spontaneous versus posed smile recognition using discriminative local spatio-temporal descriptors . . Proc. Int. IEEE Conf. on Acoustics, Speech and Signal Processing , , 2014 . . p.1249 - - 1253 . . DOI: 10.1109/ICASSP.2014.6853795 http://doi.org/10.1109/ICASSP.2014.6853795 . .

Q. Wu , , , X.B. Shen , , , X.L. Fu . . The machine knows what you are hiding: an automatic micro-expression recognition system . . LNCS , , 2011 . . 6975 152 - - 162 . . DOI: 10.1007/978-3-642-24571-8_16 http://doi.org/10.1007/978-3-642-24571-8_16 . .

J.C. Yang , , , K. Yu , , , Y.H. Gong , , , 等 . . Linear spatial pyramid matching using sparse coding for image classification . . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , , 2009 . . p.1794 - - 1801 . . DOI: 10.1109/CVPR.2009.5206757 http://doi.org/10.1109/CVPR.2009.5206757 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰