基于心理生理信号的个人音乐情感体验识别

张乐凯; 孙守迁; 邢白夕; 罗睿铭; 张克俊

doi:10.1631/FITEE.1800101

Your Location：

Home >

Browse articles >

基于心理生理信号的个人音乐情感体验识别

常规文章 | Updated：2022-05-19

- 基于心理生理信号的个人音乐情感体验识别
- Using psychophysiological measures to recognize personal music emotional experience
- 信息与电子工程前沿（英文） 2019年20卷第7期页码：964-974
- Affiliations：
  
  College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  School of Design, Zhejiang University of Technology, Hangzhou 310023, China
- Author bio：
  
  [ "Le-kai ZHANG, E-mail: zlkzhang@zju.edu.cn" ]
  [ "Shou-qian SUN, E-mail: ssq@zju.edu.cn" ]
  Bai-xi XING, E-mail: sisyxing@gmail.com
  [ "Rui-ming LUO, E-mail: joeluo@zju.edu.cn" ]
- Funds：
  
  Project supported by the Philosophy and Social Science Planning Fund Project of Zhejiang Province, China (No. 20NDQN297YB) and the National Natural Science Foundation of China (No. 61702454)
- DOI：10.1631/FITEE.1800101
  中图分类号： TP391.4
- 收稿：2018-02-09，
  
  修回：2019-;7-12，
  
  纸质出版：2019-07
- Accepted：
Scan QR Code
张乐凯, 孙守迁, 邢白夕, 等. 基于心理生理信号的个人音乐情感体验识别[J]. 信息与电子工程前沿（英文）, 2019,20(7):964-974.

Le-kai ZHANG, Shou-qian SUN, Bai-xi XING, et al. Using psychophysiological measures to recognize personal music emotional experience[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(7): 964-974.
张乐凯, 孙守迁, 邢白夕, 等. 基于心理生理信号的个人音乐情感体验识别[J]. 信息与电子工程前沿（英文）, 2019,20(7):964-974. DOI： 10.1631/FITEE.1800101.

Le-kai ZHANG, Shou-qian SUN, Bai-xi XING, et al. Using psychophysiological measures to recognize personal music emotional experience[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(7): 964-974. DOI： 10.1631/FITEE.1800101.

摘要

音乐能激发人的情感，这是一个心理生理过程。因此心理生理特征可用于识别个人音乐情感体验。提出一种新的基于人体生理特征的个人音乐情感识别方法。首先，建立一个基于音乐情感特征的数据库和一个基于听音乐产生的生理信号的数据库，包括心电、脉搏、皮温、呼吸和瞳孔直径变化等生理信号。然后，分别采用线性回归、岭回归、三种不同核的支持向量机、决策树、

近邻算法、多层感知器和Nu支持向量回归（NuSVR）方法，通过音乐特征和人体生理特征识别音乐情感。结果显示，NuSVR性能优于其他方法，其唤醒相关系数为0.7347（均方差为0.023 23），效价相关系数为0.7902（均方差为0.014 85）。最后，对不同数据集进行比较，结果表明所有特征（音乐特征和所有生理特征）数据集在识别中表现最好，唤醒相关系数为0.6499（均方差为0.029 32），效价相关系数为0.7735（均方差为0.015 76）。本文提供了一种有效的个人音乐情感体验识别方法，可用于个性化音乐推荐。

Abstract

Music can trigger human emotion. This is a psychophysiological process. Therefore

using psychophysiological characteristics could be a way to understand individual music emotional experience. In this study

we explore a new method of personal music emotion recognition based on human physiological characteristics. First

we build up a database of features based on emotions related to music and a database based on physiological signals derived from music listening including EDA

PPG

SKT

RSP

and PD variation information. Then linear regression

ridge regression

support vector machines with three different kernels

decision trees

$$k$$

-nearest neighbors

multi-layer perceptron

and Nu support vector regression (NuSVR) are used to recognize music emotions via a data synthesis of music features and human physiological features. NuSVR outperforms the other methods. The correlation coefficient values are 0.7347 for arousal and 0.7902 for valence

while the mean squared errors are 0.023 23 for arousal and 0.014 85 for valence. Finally

we compare the different data sets and find that the data set with all the features (music features and all physiological features) has the best performance in modeling. The correlation coefficient values are 0.6499 for arousal and 0.7735 for valence

while the mean squared errors are 0.029 32 for arousal and 0.015 76 for valence. We provide an effective way to recognize personal music emotional experience

and the study can be applied to personalized music recommendation.

关键词

Keywords

references

F Agrafioti , , , D Hatzinakos , , , AK Anderson . . ECG pattern analysis for emotion detection . . IEEE Trans Affect Comput , , 2012 . . 3 ( ( 1 ): ): 102 - - 115 . . DOI: 10.1109/T-AFFC.2011.28 http://doi.org/10.1109/T-AFFC.2011.28 . .

ME Ayadi , , , MS Kamel , , , F Karray . . Survey on speech emotion recognition: features, classification schemes, and databases . . Patt Recogn , , 2011 . . 44 ( ( 3 ): ): 572 - - 587 . . DOI: 10.1016/j.patcog.2010.09.020 http://doi.org/10.1016/j.patcog.2010.09.020 . .

MM Bradley , , , PJ Lang . . Measuring emotion: the selfassessment manikin and the semantic differential . . J Behav Ther Exp Psych , , 1994 . . 25 ( ( 1 ): ): 49 - - 59 . . DOI: 10.1016/0005-7916(94)90063-9 http://doi.org/10.1016/0005-7916(94)90063-9 . .

C Chandler , , , R Cornes . . Biometric measurement of human emotions . . TECHNIA - Int J Comput Sci Commun Technol , , 2012 . . 4 ( ( 2 ): ): 164 - - 168 . . . .

B Cheng , , , GY Liu . . Emotion recognition from surface EMG signal using wavelet transform and neural network . . J Comput Appl , , 2008 . . 28 ( ( 2 ): ): 333 - - 335 . . . .

NAJ de Witte , , , S Sütterlin , , , C Braet , , , 等 . . Psychophysiological correlates of emotion regulation training in adolescent anxiety: evidence from the novel PIER task . . J Affect Disord , , 2017 . . 214 89 - - 96 . . DOI: 10.1016/j.jad.2017.03.012 http://doi.org/10.1016/j.jad.2017.03.012 . .

ABM Gerdes , , , MJ Wieser , , , GW Alpers . . Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains . . Front Psychol , , 2014 . . 5 1351 DOI: 10.3389/fpsyg.2014.01351 http://doi.org/10.3389/fpsyg.2014.01351 . .

CD Katsis , , , NS Katertsidis , , , DI Fotiadis . . An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders . . Biomed Signal Process Contr , , 2011 . . 6 ( ( 3 ): ): 261 - - 268 . . DOI: 10.1016/j.bspc.2010.12.001 http://doi.org/10.1016/j.bspc.2010.12.001 . .

J Kim , , , E André . . Emotion recognition based on physiological changes in music listening . . IEEE Trans Patt Anal Mach Intell , , 2008 . . 30 ( ( 12 ): ): 2067 - - 2083 . . DOI: 10.1109/TPAMI.2008.26 http://doi.org/10.1109/TPAMI.2008.26 . .

S Koelstra , , , C Muhl , , , M Soleymani , , , 等 . . DEAP: a database for emotion analysis; using physiological signals . . IEEE Trans Affect Comput , , 2012 . . 3 ( ( 1 ): ): 18 - - 31 . . DOI: 10.1109/T-AFFC.2011.15 http://doi.org/10.1109/T-AFFC.2011.15 . .

CL Krumhansl . . An exploratory study of musical emotions and psychophysiology . . Can J Exp Psychol , , 1997 . . 51 ( ( 4 ): ): 336 - - 353 . . DOI: 10.1037/1196-1961.51.4.336 http://doi.org/10.1037/1196-1961.51.4.336 . .

C Li , , , C Xu , , , ZY Feng . . Analysis of physiological for emotion recognition with the IRS model . . Neurocomputing , , 2016 . . 178 103 - - 111 . . DOI: 10.1016/j.neucom.2015.07.112 http://doi.org/10.1016/j.neucom.2015.07.112 . .

SS Li , , , G Walters , , , J Packer , , , 等 . . A comparative analysis of self-report and psychophysiological measures of emotion in the context of tourism advertising . . J Travel Res , , 2017 . . 57 ( ( 8 ): ): 1078 - - 1092 . . DOI: 10.1177/0047287517733555 http://doi.org/10.1177/0047287517733555 . .

Q Lu , , , XO Chen , , , DS Yang , , , 等 . . Boosting for multimodal music emotion classification . . 11 th Int Society for Music Information Retrieval Conf , , 2010 . . p.105 - - 110 . . . .

CLB Maia , , , ES Furtado . . A study about psychophysiological measures in user experience monitoring and evaluation . . Proc 15 th Brazilian Symp on Human Factors in Computing Systems , , 2016 . . p.7 DOI: 10.1145/3033701.3033708 http://doi.org/10.1145/3033701.3033708 . .

MT Mitterschiffthaler , , , CHY Fu , , , JA Dalton , , , 等 . . A functional MRI study of happy and sad affective states induced by classical music . . Hum Brain Mapp , , 2007 . . 28 ( ( 11 ): ): 1150 - - 1162 . . DOI: 10.1002/hbm.20337 http://doi.org/10.1002/hbm.20337 . .

K Mori , , , M Iwanaga . . Two types of peak emotional responses to music: the psychophysiology of chills and tears . . Sci Rep , , 2017 . . 7 46063 DOI: 10.1038/srep46063 http://doi.org/10.1038/srep46063 . .

I Nyklíček , , , JF Thayer , , , LJP van Doornen . . Cardiorespiratory differentiation of musically-induced emotions . . J Psychophysiol , , 1997 . . 11 ( ( 4 ): ): 304 - - 321 . . . .

F Pedregosa , , , G Varoquaux , , , A Gramfort , , , 等 . . Scikitlearn: machine learning in Python . . J Mach Learn Res , , 2011 . . 12 ( ( 10 ): ): 2825 - - 2830 . . . .

RW Picard , , , E Vyzas , , , J Healey . . Toward machine emotional intelligence: analysis of affective physiological state . . IEEE Trans Patt Anal Mach Intell , , 2001 . . 23 ( ( 10 ): ): 1175 - - 1191 . . DOI: 10.1109/34.954607 http://doi.org/10.1109/34.954607 . .

P Rani , , , C Liu , , , N Sarkar , , , 等 . . An empirical study of machine learning techniques for affect recognition in human-robot interaction . . Patt Anal Appl , , 2006 . . 9 ( ( 1 ): ): 58 - - 69 . . DOI: 10.1007/s10044-006-0025-y http://doi.org/10.1007/s10044-006-0025-y . .

P Ren , , , A Barreto , , , Y Gao , , , 等 . . Affective assessment by digital processing of the pupil diameter . . IEEE Trans Affect Comput , , 2013 . . 4 ( ( 1 ): ): 2 - - 14 . . DOI: 10.1109/T-AFFC.2012.25 http://doi.org/10.1109/T-AFFC.2012.25 . .

G Tzanetakis , , , P Cook . . MARSYAS: a framework for audio analysis . . Organ Sound , , 2000 . . 4 ( ( 3 ): ): 169 - - 175 . . DOI: 10.1017/S1355771800003071 http://doi.org/10.1017/S1355771800003071 . .

J Wagner , , , J Kim , , , E Andre . . From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification . . IEEE Int Conf on Multimedia and Expo , , 2005 . . p.940 - - 943 . . DOI: 10.1109/ICME.2005.1521579 http://doi.org/10.1109/ICME.2005.1521579 . .

SH Wang , , , P Phillips , , , ZC Dong , , , 等 . . Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm . . Neurocomputing , , 2018 . . 272 668 - - 676 . . DOI: 10.1016/j.neucom.2017.08.015 http://doi.org/10.1016/j.neucom.2017.08.015 . .

WS Wen , , , GY Liu , , , NP Cheng , , , 等 . . Emotion recognition based on multi-variant correlation of physiological signals . . IEEE Trans Affect Comput , , 2014 . . 5 ( ( 2 ): ): 126 - - 140 . . DOI: 10.1109/TAFFC.2014.2327617 http://doi.org/10.1109/TAFFC.2014.2327617 . .

BX Xing , , , KJ Zhang , , , SQ Sun , , , 等 . . Emotion-driven Chinese folk music-image retrieval based on DE-SVM . . Neurocomputing , , 2015 . . 148 619 - - 627 . . DOI: 10.1016/j.neucom.2014.08.007 http://doi.org/10.1016/j.neucom.2014.08.007 . .

YH Yang , , , HH Chen . . Prediction of the distribution of perceived music emotions using discrete samples . . IEEE Trans Audio Speech Lang Process , , 2011 . . 19 ( ( 7 ): ): 2184 - - 2196 . . DOI: 10.1109/TASL.2011.2118752 http://doi.org/10.1109/TASL.2011.2118752 . .

T Yun , , , L Guan . . Human emotional state recognition using real 3D visual features from Gabor library . . Patt Recogn , , 2013 . . 46 ( ( 2 ): ): 529 - - 538 . . DOI: 10.1016/j.patcog.2012.08.002 http://doi.org/10.1016/j.patcog.2012.08.002 . .

YD Zhang , , , ZJ Yang , , , HM Lu , , , 等 . . Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation . . IEEE Access , , 2016 . . 4 8375 - - 8385 . . DOI: 10.1109/ACCESS.2016.2628407 http://doi.org/10.1109/ACCESS.2016.2628407 . .

X Zhu . . Emotion recognition of EMG based on BP neural network . . Proc 2 nd Int Symp on Networking and Network Security , , 2010 . . p.227 - - 229 . . . .

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

A personality-guided affective brain–computer interface for implementation of emotional intelligence in machines

Using Kinect for real-time emotion recognition via facial expressions