Using Kinect for real-time emotion recognition via facial expressions

MAO Qi-rong; PAN Xin-yu; ZHAN Yong-zhao; SHEN Xiang-jun

doi:10.1631/FITEE.1400209

Your Location：

Home >

Browse articles >

Using Kinect for real-time emotion recognition via facial expressions

Computer & Automation | Updated：2022-06-14

- Using Kinect for real-time emotion recognition via facial expressions
- 基于Kinect的实时面部情感识别
- Frontiers of Information Technology & Electronic Engineering Vol. 16, Issue 4, Pages: 272-282(2015)
- Affiliations：
  
  Department of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China
- Author bio：
  
  Qi-rong MAO, E-mail: mao_qr@ujs.edu.cn
  [ "Xin-yu PAN, E-mail: pxyz@vip.qq.com" ]
- Funds：
- DOI：10.1631/FITEE.1400209
  CLC： TP391.4
- Received：12 June 2014，
  
  Revised：2015-03-09，
  
  Published：2015-04
- Accepted：
Scan QR Code
MAO Qi-rong, PAN Xin-yu, ZHAN Yong-zhao, et al. Using Kinect for real-time emotion recognition via facial expressions[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(4): 272-282.
DOI：

MAO Qi-rong, PAN Xin-yu, ZHAN Yong-zhao, et al. Using Kinect for real-time emotion recognition via facial expressions[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(4): 272-282. DOI： 10.1631/FITEE.1400209.

摘要

目的

基于Kinect提取面部表情特征，实现对视频中面部情感实时识别。

创新点

基于Kinect提出一种新颖的面部情感识别方法，充分发挥出Kinect高速易用的优势，可实时对视频序列最新连续帧中表现出的情感进行综合判定。识别方法涉及两种不同形式的表情特征，对此我们也针对性地提出基于最大置信的融合算法。

方法

首先，运用Kinect中Face Tracking SDK从实时视频数据中追踪人脸、提取面部运动单元信息和特征点坐标（图3、4）。然后，将这两类特征信息并行处理，在它们各自特征通道中，特征数据经7元1-vs-1分类器组进行预识别，将得到的预识别结果存入缓存用于情感置信统计，置信度最高的即为此通道中的情感识别结果（图2）。最后，融合这两个特征通道的结果即可得到最终情感识别结果（图1）。

结论

基于Kinect提取到的两种面部表情特征（面部运动单元信息和特征点坐标），提出一种新颖高效的面部情感识别方法，实现对视频中面部情感实时识别。

Abstract

Emotion recognition via facial expressions (ERFE) has attracted a great deal of interest with recent advances in artificial intelligence and pattern recognition. Most studies are based on 2D images

and their performance is usually computationally expensive. In this paper

we propose a real-time emotion recognition approach based on both 2D and 3D facial expression features captured by Kinect sensors. To capture the deformation of the 3D mesh during facial expression

we combine the features of animation units (AUs) and feature point positions (FPPs) tracked by Kinect. A fusion algorithm based on improved emotional profiles (IEPs) and maximum confidence is proposed to recognize emotions with these real-time facial expression features. Experiments on both an emotion dataset and a real-time video show the superior performance of our method.

关键词

Keywords

references

J. Ahlberg . Candide-3-an Updated Parameterised Face . Technical Report , 2001 .

M. Breidt , H.H. Biilthoff , C. Curio . Robust semantic analysis by synthesis of 3D facial motion . Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition and Workshops , 2011 . p.713 - 719 . DOI: 10.1109/FG.2011.5771336 http://doi.org/10.1109/FG.2011.5771336 .

C. Cao , Y.L. Weng , S. Zhou , 等 . FaceWarehouse: a 3D facial expression database for visual computing . IEEE Trans. Visual. Comput. Graph. , 2014 . 20 ( 3 ): 413 - 425 . DOI: 10.1109/TVCG.2013.249 http://doi.org/10.1109/TVCG.2013.249 .

C.C. Chang , C.J. Lin . LIBSVM: a Library for Support Vector Machines . Available from , http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html http://www.csie.ntu.edu.tw/~cjlin/libsvm/index.html , 2011a .

C.C. Chang , C.J. Lin . LIBSVM: a library for support vector machines . ACM Trans. Intell. Syst. Technol. , 2011b . 2 ( 3 ): 27 DOI: 10.1145/1961189.1961199 http://doi.org/10.1145/1961189.1961199 .

D. Cosker , E. Krumhuber , A. Hilton . A FACS valid 3D dynamic action unit database with applications to 3D dynamic morphable facial modeling . Proc. IEEE Int. Conf. on Computer Vision , 2011 . p.2296 - 2303 . DOI: 10.1109/ICCV.2011.6126510 http://doi.org/10.1109/ICCV.2011.6126510 .

P. Ekman . Facial expression and emotion . Am. Psychol. , 1993 . 48 ( 4 ): 384 - 392 . DOI: 10.1037/0003-066X.48.4.384 http://doi.org/10.1037/0003-066X.48.4.384 .

P. Ekman , W.V. Friesen . Facial action coding system: a technique for the measurement of facial movement . Palo Alto , 1978 .

B. Fasel , J. Luettin . Automatic facial expression analysis: a survey . Patt. Recog. , 2003 . 36 ( 1 ): 259 - 275 . DOI: 10.1016/S0031-3203(02)00052-3 http://doi.org/10.1016/S0031-3203(02)00052-3 .

R.I. Hg , P. Jasek , C. Rofidal , 等 . An RGB-D database using Microsoft's Kinect for windows for face detection . Proc. 8th Int. Conf. on Signal Image Technology and Internet Based Systems , 2012 . p.42 - 46 . DOI: 10.1109/SITIS.2012.17 http://doi.org/10.1109/SITIS.2012.17 .

B.Y. Li , A.S. Mian , W.Q. Liu , 等 . Using Kinect for face recognition under varying poses, expressions, illumination and disguise . Proc. IEEE Workshop on Applications of Computer Vision , 2013 . p.186 - 192 . DOI: 10.1109/WACV.2013.6475017 http://doi.org/10.1109/WACV.2013.6475017 .

D.X. Li , C. Sun , F.Q. Hu , 等 . Real-time performance-driven facial animation with 3ds Max and Kinect . Proc. 3rd Int. Conf. on Consumer Electronics, Communications and Networks , 2013 . p.473 - 476 . DOI: 10.1109/CECNet.2013.6703372 http://doi.org/10.1109/CECNet.2013.6703372 .

X.H. Ma , Y.Q. Tan , G.M. Zheng . A fast classification scheme and its application to face recognition . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , 2013 . 14 ( 7 ): 561 - 572 . DOI: 10.1631/jzus.CIDE1309 http://doi.org/10.1631/jzus.CIDE1309 .

Q.R. Mao , X.L. Zhao , Z.W. Huang , 等 . Speakerindependent speech emotion recognition by fusion of functional and accompanying paralanguage features . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , 2013 . 14 ( 7 ): 573 - 582 . DOI: 10.1631/jzus.CIDE1310 http://doi.org/10.1631/jzus.CIDE1310 .

M.A. Nicolaou , H. Gunes , M. Pantic . Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space . IEEE Trans. Affect. Comput. , 2011 . 2 ( 2 ): 92 - 105 . DOI: 10.1109/T-AFFC.2011.9 http://doi.org/10.1109/T-AFFC.2011.9 .

A. Savran , N. Alyüz , H. Dibeklioğlu , 等 . Bosphorus database for 3D face analysis . Proc. 1st European Workshop on Biometrics and Identity Management , 2008 . p.47 - 56 . DOI: 10.1007/978-3-540-89991-4_6 http://doi.org/10.1007/978-3-540-89991-4_6 .

B. Seddik , H. Maamatou , S. Gazzah , 等 . Unsupervised facial expressions recognition and avatar reconstruction from Kinect . Proc. 10th Int. Multi-conf. on Systems, Signals & Devices , 2013 . p.1 - 6 . DOI: 10.1109/SSD.2013.6564032 http://doi.org/10.1109/SSD.2013.6564032 .

G. Stratou , A. Ghosh , P. Debevec , 等 . Effect of illumination on automatic expression recognition: a novel 3D relightable facial database . Proc. IEEE Int. Conf. on Automatic Face & Gesture Recognition and Workshops , 2011 . p.611 - 618 . DOI: 10.1109/FG.2011.5771467 http://doi.org/10.1109/FG.2011.5771467 .

Hurk Y. van den . Gender Classification with Visual and Depth Images . MS Thesis, Tilburg University, the Netherlands , 2012 .

A. Vinciarelli , M. Pantic , D. Heylen , 等 . Bridging the gap between social animal and unsocial machine: a survey of social signal processing . IEEE Trans. Affect. Comput. , 2012 . 3 ( 1 ): 69 - 87 . DOI: 10.1109/T-AFFC.2011.27 http://doi.org/10.1109/T-AFFC.2011.27 .

S.B. Xu , G.H. Ma , W.L. Meng , 等 . Statistical learning based facial animation . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , 2013 . 14 ( 7 ): 542 - 550 . DOI: 10.1631/jzus.CIDE1307 http://doi.org/10.1631/jzus.CIDE1307 .

Z. Zeng , M. Pantic , G.I. Roisman , 等 . A survey of affect recognition methods: audio, visual, and spontaneous expressions . IEEE Trans. Patt. Anal. Mach. Intell. , 2009 . 31 ( 1 ): 39 - 58 . DOI: 10.1109/TPAMI.2008.52 http://doi.org/10.1109/TPAMI.2008.52 .

X.X. Zhu , D. Ramanan . Face detection, pose estimation, and landmark localization in the wild . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , 2012 . p.2879 - 2886 . DOI: 10.1109/CVPR.2012.6248014 http://doi.org/10.1109/CVPR.2012.6248014 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

A personality-guided affective brain–computer interface for implementation of emotional intelligence in machines

An efficient parallel and distributed solution to nonconvex penalized linear SVMs

Using psychophysiological measures to recognize personal music emotional experience

A robust intelligent audio watermarking scheme using support vector machine

Performance analysis of new word weighting procedures for opinion mining

Related Author

Bin HU

Xiaowei ZHANG

Xiangyu WEI

Wenjie YUAN

Zejian XING

Wei LI

Shaojie LI

SUN Tao

Related Institution

School of Information Science and Engineering, Lanzhou University

College of Computer, National University of Defense Technology

School of Computer Science, Fudan University

Shanghai Key Laboratory of Data Science

College of Computer Science and Technology, Zhejiang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰