Affective rating ranking based on face images in arousal-valence dimensional space

Guo-peng XU; Hai-tang LU; Fei-fei ZHANG; Qi-rong MAO

doi:10.1631/FITEE.1700270

Your Location：

Home >

Browse articles >

Affective rating ranking based on face images in arousal-valence dimensional space

Regular Papers | Updated：2022-05-19

- Affective rating ranking based on face images in arousal-valence dimensional space
- 激活-效价维度空间中基于脸部图像的情感等级排序方法
- Frontiers of Information Technology & Electronic Engineering Vol. 19, Issue 6, Pages: 783-795(2018)
- Affiliations：
  
  School of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China
- Author bio：
  
  [ "Guo-peng XU, gpxu@ujs.edu.cn" ]
  [ "Hai-tang LU, 1406404872@qq.com" ]
  [ "Fei-fei ZHANG, susanzhang1231@sina.com" ]
  Qi-rong MAO, E-mail:mao_qr@mail.ujs.edu.cn
- Funds：
- DOI：10.1631/FITEE.1700270
  CLC： TP391
- Received：19 April 2017，
  
  Revised：2018-;6-12，
  
  Published：2018-06
- Accepted：
Scan QR Code
Guo-peng XU, Hai-tang LU, Fei-fei ZHANG, et al. Affective rating ranking based on face images in arousal-valence dimensional space[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(6): 783-795.
DOI：

Guo-peng XU, Hai-tang LU, Fei-fei ZHANG, et al. Affective rating ranking based on face images in arousal-valence dimensional space[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(6): 783-795. DOI： 10.1631/FITEE.1700270.

摘要

在维度情感识别领域，分类和回归通常被用来对情感的机器学习进行建模和预测。然而，在维度情感空间中，情感标注通常是一个连续的实数值，拥有有序属性。而前面所提的两种方法并没有考虑并利用这一重要信息。因此，我们提出一个在激活和效价维度空间下的基于脸部图像的情感等级排序框架。我们的方法能够通过离散化连续的情感标注得到情感等级，并恰当地利用它们之间的有序信息。确切地说，首先训练一系列基本误差敏感二分类器，每个二分类器都使用经过二值重新标注的全部样本。依据样本对应的情感等级与给定二分类器对应的情感等级的比较结果，对二值进行重新标注。然后通过聚合所有二分类器的输出结果，可以得到样本最终的情感等级预测结果。在AVEC 2015挑战赛标准数据集和SEMAINE子集数据集上对所提方法与基本的和基于深度学习的分类和回归方法进行比较。实验结果表明，所提出的基于排序的情感识别方法在激活和效价两个维度上都是有效的。

Abstract

In dimensional affect recognition

the machine learning methods

which are used to model and predict affect

are mostly classification and regression. However

the annotation in the dimensional affect space usually takes the form of a continuous real value which has an ordinal property. The aforementioned methods do not focus on taking advantage of this important information. Therefore

we propose an affective rating ranking framework for affect recognition based on face images in the valence and arousal dimensional space. Our approach can appropriately use the ordinal information among affective ratings which are generated by discretizing continuous annotations. Specifically

we first train a series of basic cost-sensitive binary classifiers

each of which uses all samples relabeled according to the comparison results between corresponding ratings and a given rank of a binary classifier. We obtain the final affective ratings by aggregating the outputs of binary classifiers. By comparing the experimental results with the baseline and deep learning based classification and regression methods on the benchmarking database of the AVEC 2015 Challenge and the selected subset of SEMAINE database

we find that our ordinal ranking method is effective in both arousal and valence dimensions.

关键词

Keywords

references

F Abousaleh , , , T Lim , , , W Cheng , , , 等 . . A novel comparative deep learning framework for facial age estimation . . EURASIP J Image Video Process , , 2016 . . 2016 ( ( 1 ): ): 47 DOI: 10.1186/s13640-016-0151-4 http://doi.org/10.1186/s13640-016-0151-4 . .

S Baron-Cohen . . Mind Reading: the Interactive Guide to Emotions . . Jessica Kingsley Publishers , , 2004 . . .

J Bruna , , , S Mallat . . Invariant scattering convolution networks . . IEEE Trans Patt Anal Mach Intell , , 2013 . . 35 ( ( 8 ): ): 1872 - - 1886 . . DOI: 10.1109/TPAMI.2012.230 http://doi.org/10.1109/TPAMI.2012.230 . .

G Caridakis , , , L Malatesta , , , L Kessous , , , 等 . . Modeling naturalistic affective states via facial and vocal expressions recognition . . 8 th Int Conf on Multim Interfaces , , 2006 . . p.146 - - 154 . . DOI: 10.1145/1180995.1181029 http://doi.org/10.1145/1180995.1181029 . .

C Chang , , , C Lin . . LIBSVM: a library for support vector machines . . ACM Trans Intell Syst Technol , , 2011 . . 2 ( ( 3 ): ): 27 DOI: 10.1145/1961189.1961199 http://doi.org/10.1145/1961189.1961199 . .

K Chang , , , C Chen . . A learning framework for age rank estimation based on face images with scattering transform . . IEEE Trans Image Process , , 2015 . . 24 ( ( 3 ): ): 785 - - 798 . . DOI: 10.1109/TIP.2014.2387379 http://doi.org/10.1109/TIP.2014.2387379 . .

K Chang , , , C Chen , , , Y Hung . . A ranking approach for human ages estimation based on face images . . 20 th Int Conf on Pattern Recognition , , 2010 . . p.3396 - - 3399 . . DOI: 10.1109/ICPR.2010.829 http://doi.org/10.1109/ICPR.2010.829 . .

S Feng , , , C Lang , , , J Feng , , , 等 . . Human facial age estimation by cost-sensitive label ranking and trace norm regularization . . IEEE Trans Multim , , 2017 . . 19 ( ( 1 ): ): 136 - - 148 . . DOI: 10.1109/TMM.2016.2608786 http://doi.org/10.1109/TMM.2016.2608786 . .

X Geng , , , Z Zhou , , , K Smith-Miles . . Automatic age estimation based on facial aging patterns . . IEEE Trans Patt Anal Mach Intell , , 2007 . . 29 ( ( 12 ): ): 2234 - - 2240 . . DOI: 10.1109/TPAMI.2007.70733 http://doi.org/10.1109/TPAMI.2007.70733 . .

D Glowinski , , , A Camurri , , , G Volpe , , , 等 . . Technique for automatic emotion recognition by body gesture analysis . . Int Conf on Computer Vision and Pattern Recognition Workshops , , 2008 . . p.1 - - 6 . . DOI: 10.1109/CVPRW.2008.4563173 http://doi.org/10.1109/CVPRW.2008.4563173 . .

H Gunes , , , M Pantic . . Automatic, dimensional and continuous emotion recognition . . Int J Synth Emot , , 2010 . . 1 ( ( 1 ): ): 68 - - 99 . . DOI: 10.4018/jse.2010101605 http://doi.org/10.4018/jse.2010101605 . .

L He , , , D Jiang , , , L Yang , , , 等 . . Multimodal affective dimension prediction using deep bidirectional long short-term memory recurrent neural networks . . 5 th Int Workshop on Audio/Visual Emotion Challenge , , 2015 . . p.73 - - 80 . . DOI: 10.1145/2808196.2811641 http://doi.org/10.1145/2808196.2811641 . .

S Ioannou , , , A Raouzaiou , , , V Tzouvaras , , , 等 . . Emotion recognition through facial expression analysis based on a neurofuzzy network . . Neur Netw , , 2005 . . 18 ( ( 4 ): ): 423 - - 435 . . DOI: 10.1016/j.neunet.2005.03.004 http://doi.org/10.1016/j.neunet.2005.03.004 . .

T Joachims . . Optimizing search engines using clickthrough data . . 8 th ACM Int Conf on Knowledge Discovery and Data Mining , , 2002 . . p.133 - - 142 . . DOI: 10.1145/775047.775067 http://doi.org/10.1145/775047.775067 . .

G Levi , , , T Hassncer . . Age and gender classification using convolutional neural networks . . Int Conf on Computer Vision and Pattern Recognition Workshops , , 2015 . . p.34 - - 42 . . DOI: 10.1109/CVPRW.2015.7301352 http://doi.org/10.1109/CVPRW.2015.7301352 . .

L Li , , , H Lin . . Ordinal regression by extended binary classification . . Advances in Neural Information Processing Systems , , 2006 . . p.865 - - 872 . . . .

T Lim , , , K Hua , , , H Wang , , , 等 . . VRank: voting system on ranking model for human age estimation . . 17 th IEEE Int Workshop on Multimedia Signal Processing , , 2015 . . p.1 - - 6 . . DOI: 10.1109/MMSP.2015.7340789 http://doi.org/10.1109/MMSP.2015.7340789 . .

T Liu . . Learning to Rank for Information Retrieval . . Springer-Verlag Berlin Heidelberg , , 2011 . . .

H Martinez , , , G Yannakakis , , , J Hallam . . Don't classify ratings of affect; rank them! . . IEEE Trans Affect Comput , , 2014 . . 5 ( ( 3 ): ): 314 - - 326 . . DOI: 10.1109/TAFFC.2014.2352268 http://doi.org/10.1109/TAFFC.2014.2352268 . .

D McDuff , , , R El Kaliouby , , , K Kassam , , , 等 . . Affect valence inference from facial action unit spectrograms . . Int Conf on Computer Vision and Pattern Recognition Workshops , , 2010 . . p.17 - - 24 . . DOI: 10.1109/CVPRW.2010.5543833 http://doi.org/10.1109/CVPRW.2010.5543833 . .

M Nicolaou , , , H Gunes , , , M Pantic . . Audio-visual classification and fusion of spontaneous affective data in likelihood space . . 20 th Int Conf on Pattern Recognition , , 2010 . . p.3695 - - 3699 . . DOI: 10.1109/ICPR.2010.900 http://doi.org/10.1109/ICPR.2010.900 . .

M Nicolaou , , , H Gunes , , , M Pantic . . Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space . . IEEE Trans Affect Comput , , 2011 . . 2 ( ( 2 ): ): 92 - - 105 . . DOI: 10.1109/T-AFFC.2011.9 http://doi.org/10.1109/T-AFFC.2011.9 . .

F Ringeval , , , A Sonderegger , , , J Sauer , , , 等 . . Introducing the RECOLA multimodal corpus of remote collaborative and affective interactions . . 10 th IEEE Int Conf on Automatic Face and Gesture Recognition Workshops , , 2013 . . p.1 - - 8 . . DOI: 10.1109/FG.2013.6553805 http://doi.org/10.1109/FG.2013.6553805 . .

F Ringeval , , , B Schuller , , , M Valstar , , , 等 . . AVEC 2015: the 5 th International Audio/Visual Emotion Challenge and Workshop . . 23 rd ACM Int Conf on Multimedia , , 2015 . . p.1335 - - 1336 . . DOI: 10.1145/2733373.2806408 http://doi.org/10.1145/2733373.2806408 . .

J Russell . . A circumplex model of affect . . J Pers Soc Psychol , , 1980 . . 39 ( ( 6 ): ): 1161 - - 1178 . . DOI: 10.1037/h0077714 http://doi.org/10.1037/h0077714 . .

K Scherer . . Psychological models of emotion . . In: Borod J (Ed.), The Neuropsychology of Emotion. Oxford University Press, New York, USA , , 2000 . . .

K Scherer , , , A Schorr , , , T Johnstone . . Appraisal Processes in Emotion: Theory, Methods, Research . . Oxford University Press, New York, USA , , 2001 . . .

B Schuller , , , B Vlasenko , , , F Eyben , , , 等 . . Acoustic emotion recognition: a benchmark comparison of performances . . IEEE Workshop on Automatic Speech Recognition and Understanding , , 2009 . . p.552 - - 557 . . DOI: 10.1109/ASRU.2009.5372886 http://doi.org/10.1109/ASRU.2009.5372886 . .

T Senechal , , , V Rapp , , , H Salam , , , 等 . . Facial action recognition combining heterogeneous features via multikernel learning . . IEEE Trans Syst Man Cybern Part B (Cybern) , , 2012 . . 42 ( ( 4 ): ): 993 - - 1005 . . DOI: 10.1109/TSMCB.2012.2193567 http://doi.org/10.1109/TSMCB.2012.2193567 . .

M Wöllmer , , , F Eyben , , , S Reiter , , , 等 . . Abandoning emotion classes—towards continuous emotion recognition with modelling of long-range dependencies . . Interspeech , , 2008 . . p.597 - - 600 . . . .

M Wöllmer , , , A Metallinou , , , F Eyben , , , 等 . . Contextsensitive multimodal emotion recognition from speech and facial expression using bidirectional LSTM modeling . . Interspeech , , 2010a . . p.2362 - - 2365 . . . .

M Wöllmer , , , B Schuller , , , F Eyben , , , 等 . . Combining long short-term memory and dynamic Bayesian networks for incremental emotion-sensitive artificial listening . . IEEE J Sel Top Signal Process , , 2010b . . 4 ( ( 5 ): ): 867 - - 881 . . DOI: 10.1109/JSTSP.2010.2057200 http://doi.org/10.1109/JSTSP.2010.2057200 . .

J Xu , , , H Li . . AdaRank: a boosting algorithm for information retrieval . . 30 th ACM Int Conf on Research and Development in Information Retrieval , , 2007 . . p.391 - - 398 . . DOI: 10.1145/1277741.1277809 http://doi.org/10.1145/1277741.1277809 . .

Y Yang , , , H Chen . . Ranking-based emotion recognition for music organization and retrieval . . IEEE Trans Audio Speech Lang Process , , 2011 . . 19 ( ( 4 ): ): 762 - - 774 . . DOI: 10.1109/TASL.2010.2064164 http://doi.org/10.1109/TASL.2010.2064164 . .

C Yu , , , P Aoki , , , A Woodruff . . Detecting user engagement in everyday conversations . . 8 th Int Conf on Spoken Language Processing , , 2004 . . p.1329 - - 1332 . . . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰