Speech emotion recognition with unsupervised feature learning

HUANG Zheng-wei; XUE Wen-tao; MAO Qi-rong

doi:10.1631/FITEE.1400323

Your Location：

Home >

Browse articles >

Speech emotion recognition with unsupervised feature learning

Computer & Automation | Updated：2022-06-06

- Speech emotion recognition with unsupervised feature learning
- 基于无监督特征学习的语音情感识别方法
- Frontiers of Information Technology & Electronic Engineering Vol. 16, Issue 5, Pages: 358-366(2015)
- Affiliations：
  
  Department of Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China
- Author bio：
  
  [ "Zheng-wei HUANG, E-mail: zhengwei.hg@gmail.com" ]
  [ "Wen-tao XUE, E-mail: striveyou@163.com" ]
  Qi-rong MAO, E-mail: mao_qr@mail.ujs.edu.cn
- Funds：
- DOI：10.1631/FITEE.1400323
  CLC： TP391.4
- Received：16 September 2014，
  
  Revised：2015-04-10，
  
  Published：2015-05
- Accepted：
Scan QR Code
HUANG Zheng-wei, XUE Wen-tao, MAO Qi-rong. Speech emotion recognition with unsupervised feature learning[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(5): 358-366.
DOI：

HUANG Zheng-wei, XUE Wen-tao, MAO Qi-rong. Speech emotion recognition with unsupervised feature learning[J]. Frontiers of Information Technology & Electronic Engineering, 2015, 16(5): 358-366. DOI： 10.1631/FITEE.1400323.

摘要

目的

语音情感识别是人机交互的关键技术之一。同时，良好的情感特征对语音情感识别系统性能具有极大影响。目前的语音情感特征主要通过手工设计方法提取，对于其是否能够很好地刻画情感特性以及是否存在最优情感特征集，相关研究者并没有达成公认。所以有必要对语音情感特征提取进行进一步深入研究。

创新点

提出一种基于数据驱动的无监督情感特征学习方法。该方法能够自动从无标注语音数据中学习产生与情感相关的特征映射函数，用于语音情感特征提取。

方法

采用三种无监督学习算法（

-均值聚类，稀疏自动编码器，稀疏受限玻尔兹曼机）从若干无标注语音块中学习产生与目标相关的特征提取器，继而对整个语音样本进行特征提取（卷积和池化），最后训练一个线性支持向量机对未知样本进行识别。同时对模型涉及的超参数（块大小和隐层结点数目）进行选择。

结论

相对于传统原始特征，学习产生的特征具有一定的稀疏性并且对说话人及其他扰动因素具有一定鲁棒性。实验结果表明，尺寸较大的块和数量较多的隐层结点有助于提升系统性能（图4、5）。

Abstract

Emotion-based features are critical for achieving high performance in a speech emotion recognition (SER) system. In general

it is difficult to develop these features due to the ambiguity of the ground-truth. In this paper

we apply several unsupervised feature learning algorithms (including

-means clustering

the sparse auto-encoder

and sparse restricted Boltzmann machines)

which have promise for learning task-related features by using unlabeled data

to speech emotion recognition. We then evaluate the performance of the proposed approach and present a detailed analysis of the effect of two important factors in the model setup

the content window size and the number of hidden layer nodes. Experimental results show that larger content windows and more hidden nodes contribute to higher performance. We also show that the two-layer network cannot explicitly improve performance compared to a single-layer network.

关键词

Keywords

references

O. Abdel-Hamid , A.R. Mohamed , H. Jiang , 等 . Applying convolutional neural networks concepts to hybrid NN-HMM model for speech recognition . IEEE Int. Conf. on Acoustics, Speech and Signal Processing , 2012 . p.4277 - 4280 . DOI: 10.1109/ICASSP.2012.6288864 http://doi.org/10.1109/ICASSP.2012.6288864 .

F. Burkhardt , A. Paeschke , M. Rolfes , 等 . A database of German emotional speech . Interspeech , 2005 . p.1517 - 1520 . .

T.H. Chan , K. Jia , S. Gao , 等 . PCANet: a simple deep learning baseline for image classification . arXiv preprint , 2014 . arXiv:1404.3606 .

A. Coates , A.Y. Ng , H. Lee . An analysis of single-layer networks in unsupervised feature learning . Int. Conf. on Artificial Intelligence and Statistics , 2011 . p.215 - 223 . .

G.E. Dahl , D. Yu , L. Deng , 等 . Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition . IEEE Trans. Audio Speech Lang. Process. , 2012 . 20 ( 1 ): 30 - 42 . DOI: 10.1109/TASL.2011.2134090 http://doi.org/10.1109/TASL.2011.2134090 .

M. El Ayadi , M.S. Kamel , F. Karray . Survey on speech emotion recognition: features, classification schemes, and databases . Patt. Recogn. , 2011 . 44 ( 3 ): 572 - 587 . DOI: 10.1016/j.patcog.2010.09.020 http://doi.org/10.1016/j.patcog.2010.09.020 .

M. Feraru , M. Zbancioc . Speech emotion recognition for SROL database using weighted KNN algorithm . Int. Conf. on Electronics, Computers and Artificial Intelligence , 2013 . p.1 - 4 . DOI: 10.1109/ECAI.2013.6636198 http://doi.org/10.1109/ECAI.2013.6636198 .

H. Gao , S.G. Chen , P. An , 等 . Emotion recognition of Mandarin speech for different speech corpora based on nonlinear features . IEEE 11th Int. Conf. on Signal Processing , 2012 . p.567 - 570 . DOI: 10.1109/ICoSP.2012.6491552 http://doi.org/10.1109/ICoSP.2012.6491552 .

H. Gunes , B. Schuller . Categorical and dimensional affect analysis in continuous input: current trends and future directions . Image Vis. Comput. , 2013 . 31 ( 2 ): 120 - 136 . DOI: 10.1016/j.imavis.2012.06.016 http://doi.org/10.1016/j.imavis.2012.06.016 .

S. Haq , P.J. Jackson . Speaker-dependent audio-visual emotion recognition . Auditory-Visual Speech Processing , 2009 . p.53 - 58 . .

G. Hinton , L. Deng , D. Yu , 等 . Deep neural networks for acoustic modeling in speech recognition: the shared views of four research groups . IEEE Signal Process. Mag. , 2012 . 29 ( 6 ): 82 - 97 . DOI: 10.1109/MSP.2012.2205597 http://doi.org/10.1109/MSP.2012.2205597 .

Y. Kim , H. Lee , E.M. Provost . Deep learning for robust feature generation in audiovisual emotion recognition . IEEE Int. Conf. on Acoustics, Speech and Signal Processing , 2013 . p.3687 - 3691 . DOI: 10.1109/ICASSP.2013.6638346 http://doi.org/10.1109/ICASSP.2013.6638346 .

S.G. Koolagudi , S. Devliyal , A. Barthwal , 等 . Emotion recognition from semi natural speech using artificial neural networks and excitation source features . In: Contemporary Computing. Springer Berlin Heidelberg , 2012 . p.273 - 282 . .

D. Le , E.M. Provost . Emotion recognition from spontaneous speech using hidden Markov models with deep belief networks . IEEE Workshop on Automatic Speech Recognition and Understanding , 2013 . p.216 - 221 . DOI: 10.1109/ASRU.2013.6707732 http://doi.org/10.1109/ASRU.2013.6707732 .

H. Lee , P. Pham , Y. Largman , 等 . Unsupervised feature learning for audio classification using convolutional deep belief networks . Advances in Neural Information Processing Systems , 2009 . p.1096 - 1104 . .

L. Li , Y. Zhao , D. Jiang , 等 . Hybrid deep neural network--hidden Markov model (DNN-HMM) based speech emotion recognition . Humaine Association Conf. on Affective Computing and Intelligent Interaction , 2013 . p.312 - 317 . DOI: 10.1109/ACII.2013.58 http://doi.org/10.1109/ACII.2013.58 .

Q. Mao , X. Wang , Y. Zhan . Speech emotion recognition method based on improved decision tree and layered feature selection . Int. J. Human. Robot. , 2010 . 7 ( 2 ): 245 - 261 . DOI: 10.1142/S0219843610002088 http://doi.org/10.1142/S0219843610002088 .

Q.R. Mao , X.L. Zhao , Z.W. Huang , 等 . Speaker-independent speech emotion recognition by fusion of functional and accompanying paralanguage features . J. Zhejiang Univ.-Sci. C (Comput. & Electron.) , 2013 . 14 ( 7 ): 573 - 582 . DOI: 10.1631/jzus.CIDE1310 http://doi.org/10.1631/jzus.CIDE1310 .

O. Martin , I. Kotsia , B. Macq , 等 . The eNTERFACE'05 audio-visual emotion database . Proc. Int. Conf. on Data Engineering Workshops , 2006 . p.8 DOI: 10.1109/ICDEW.2006.145 http://doi.org/10.1109/ICDEW.2006.145 .

A. Mencattini , E. Martinelli , G. Costantini , 等 . Speech emotion recognition using amplitude modulation parameters and a combined feature selection procedure . Knowl.-Based Syst. , 2014 . 63 68 - 81 . DOI: 10.1016/j.knosys.2014.03.019 http://doi.org/10.1016/j.knosys.2014.03.019 .

A.R. Mohamed , G.E. Dahl , G. Hinton . Acoustic modeling using deep belief networks . IEEE Trans. Audio Speech Lang. Process. , 2012 . 20 ( 1 ): 14 - 22 . DOI: 10.1109/TASL.2011.2109382 http://doi.org/10.1109/TASL.2011.2109382 .

M.A. Nicolaou , H. Gunes , M. Pantic . Continuous prediction of spontaneous affect from multiple cues and modalities in valence-arousal space . IEEE Trans. Affect. Comput. , 2011 . 2 ( 2 ): 92 - 105 . DOI: 10.1109/T-AFFC.2011.9 http://doi.org/10.1109/T-AFFC.2011.9 .

M. Pantic , A. Nijholt , A. Pentland , 等 . Human-centred intelligent human? Computer interaction (HCI2): how far are we from attaining it? . Int. J. Auton. Adapt. Commun. Syst. , 2008 . 1 ( 2 ): 168 - 187 . DOI: 10.1504/IJAACS.2008.019799 http://doi.org/10.1504/IJAACS.2008.019799 .

S. Ramakrishnan , I.M. El Emary . Speech emotion recognition approaches in human computer interaction . Telecommun. Syst. , 2013 . 52 ( 3 ): 1467 - 1478 . DOI: 10.1007/s11235-011-9624-z http://doi.org/10.1007/s11235-011-9624-z .

M. Ranzato , F.J. Huang , Y.L. Boureau , 等 . Unsupervised learning of invariant feature hierarchies with applications to object recognition . IEEE Conf. on Computer Vision and Pattern Recognition , 2007 . p.1 - 8 . DOI: 10.1109/CVPR.2007.383157 http://doi.org/10.1109/CVPR.2007.383157 .

A.S. Razavian , H. Azizpour , J. Sullivan , 等 . CNN features off-the-shelf: an astounding baseline for recognition . arXiv preprint , 2014 . arXiv:1403.6382 .

E.M. Schmidt , Y.E. Kim . Learning emotion-based acoustic features with deep belief networks . IEEE Workshop on Applications of Signal Processing to Audio and Acoustics , 2011 . p.65 - 68 . DOI: 10.1109/ASPAA.2011.6082328 http://doi.org/10.1109/ASPAA.2011.6082328 .

A. Stuhlsatz , C. Meyer , F. Eyben , 等 . Deep neural networks for acoustic emotion recognition: raising the benchmarks . IEEE Int. Conf. on Acoustics, Speech and Signal Processing , 2011 . p.5688 - 5691 . DOI: 10.1109/ICASSP.2011.5947651 http://doi.org/10.1109/ICASSP.2011.5947651 .

R. Sun , E.II Moore . Investigating glottal parameters and Teager energy operators in emotion recognition . LNCS , 2011 . 6975 425 - 434 . DOI: 10.1007/978-3-642-24571-8_54 http://doi.org/10.1007/978-3-642-24571-8_54 .

Y. Sun , X. Wang , X. Tang . Deep learning face representation from predicting 10, 000 classes . Proc. IEEE Conf. on Computer Vision and Pattern Recognition , 2013 . p.1891 - 1898 . DOI: 10.1109/CVPR.2014.244 http://doi.org/10.1109/CVPR.2014.244 .

N. Thapliyal , G. Amoli . Speech based emotion recognition with Gaussian mixture model . Int. J. Adv. Res. Comput. Eng. Technol. , 2012 . 1 ( 5 ): 65 - 69 . .

C.H. Wu , W.B. Liang . Emotion recognition of affective speech based on multiple classifiers using acoustic-prosodic information and semantic labels . IEEE Trans. Affect. Comput. , 2011 . 2 ( 1 ): 10 - 21 . DOI: 10.1109/T-AFFC.2010.16 http://doi.org/10.1109/T-AFFC.2010.16 .

S. Wu , T.H. Falk , W.Y. Chan . Automatic speech emotion recognition using modulation spectral features . Speech Commun. , 2011 . 53 ( 5 ): 768 - 785 . DOI: 10.1016/j.specom.2010.08.013 http://doi.org/10.1016/j.specom.2010.08.013 .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Neural mesh refinement

Accurate estimation of 6-DoF tooth pose in 3D intraoral scans for dental applications using deep learning

Attention-based efficient robot grasp detection network

Adaptive neural network based boundary control of a flexible marine riser system with output constraints

Adaptive tracking control of high-order MIMO nonlinear systems with prescribed performance

Related Author

Yiyi LIAO

Lu YU

Xiang GAO

Zhiwei ZHU

Wanghui DING

Kaiwei SUN

Mengfei YU

Hangzheng LIN

Related Institution

Zhejiang Provincial Key Laboratory of Information Processing, Communication and Networking (IPCAN)

College of Information Science and Electronic Engineering, Zhejiang University

Hangzhou Dental Hospital

Angel Align Inc

Zhejiang University–University of Illinois at Urbana-Champaign Institute, Zhejiang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰