Using psychophysiological measures to recognize personal music emotional experience

Le-kai ZHANG; Shou-qian SUN; Bai-xi XING; Rui-ming LUO; Ke-jun ZHANG

doi:10.1631/FITEE.1800101

Your Location：

Home >

Browse articles >

Using psychophysiological measures to recognize personal music emotional experience

Regular Papers | Updated：2022-05-19

- Using psychophysiological measures to recognize personal music emotional experience
- 基于心理生理信号的个人音乐情感体验识别
- Frontiers of Information Technology & Electronic Engineering Vol. 20, Issue 7, Pages: 964-974(2019)
- Affiliations：
  
  College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  School of Design, Zhejiang University of Technology, Hangzhou 310023, China
- Author bio：
  
  [ "Le-kai ZHANG, E-mail: zlkzhang@zju.edu.cn" ]
  [ "Shou-qian SUN, E-mail: ssq@zju.edu.cn" ]
  Bai-xi XING, E-mail: sisyxing@gmail.com
  [ "Rui-ming LUO, E-mail: joeluo@zju.edu.cn" ]
- Funds：
- DOI：10.1631/FITEE.1800101
  CLC： TP391.4
- Received：09 February 2018，
  
  Revised：2019-;7-12，
  
  Published：2019-07
- Accepted：
Scan QR Code
Le-kai ZHANG, Shou-qian SUN, Bai-xi XING, et al. Using psychophysiological measures to recognize personal music emotional experience[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(7): 964-974.
DOI：

Le-kai ZHANG, Shou-qian SUN, Bai-xi XING, et al. Using psychophysiological measures to recognize personal music emotional experience[J]. Frontiers of Information Technology & Electronic Engineering, 2019, 20(7): 964-974. DOI： 10.1631/FITEE.1800101.

摘要

音乐能激发人的情感，这是一个心理生理过程。因此心理生理特征可用于识别个人音乐情感体验。提出一种新的基于人体生理特征的个人音乐情感识别方法。首先，建立一个基于音乐情感特征的数据库和一个基于听音乐产生的生理信号的数据库，包括心电、脉搏、皮温、呼吸和瞳孔直径变化等生理信号。然后，分别采用线性回归、岭回归、三种不同核的支持向量机、决策树、

近邻算法、多层感知器和Nu支持向量回归（NuSVR）方法，通过音乐特征和人体生理特征识别音乐情感。结果显示，NuSVR性能优于其他方法，其唤醒相关系数为0.7347（均方差为0.023 23），效价相关系数为0.7902（均方差为0.014 85）。最后，对不同数据集进行比较，结果表明所有特征（音乐特征和所有生理特征）数据集在识别中表现最好，唤醒相关系数为0.6499（均方差为0.029 32），效价相关系数为0.7735（均方差为0.015 76）。本文提供了一种有效的个人音乐情感体验识别方法，可用于个性化音乐推荐。

Abstract

Music can trigger human emotion. This is a psychophysiological process. Therefore

using psychophysiological characteristics could be a way to understand individual music emotional experience. In this study

we explore a new method of personal music emotion recognition based on human physiological characteristics. First

we build up a database of features based on emotions related to music and a database based on physiological signals derived from music listening including EDA

PPG

SKT

RSP

and PD variation information. Then linear regression

ridge regression

support vector machines with three different kernels

decision trees

$$k$$

-nearest neighbors

multi-layer perceptron

and Nu support vector regression (NuSVR) are used to recognize music emotions via a data synthesis of music features and human physiological features. NuSVR outperforms the other methods. The correlation coefficient values are 0.7347 for arousal and 0.7902 for valence

while the mean squared errors are 0.023 23 for arousal and 0.014 85 for valence. Finally

we compare the different data sets and find that the data set with all the features (music features and all physiological features) has the best performance in modeling. The correlation coefficient values are 0.6499 for arousal and 0.7735 for valence

while the mean squared errors are 0.029 32 for arousal and 0.015 76 for valence. We provide an effective way to recognize personal music emotional experience

and the study can be applied to personalized music recommendation.

关键词

Keywords

references

F Agrafioti , , , D Hatzinakos , , , AK Anderson . . ECG pattern analysis for emotion detection . . IEEE Trans Affect Comput , , 2012 . . 3 ( ( 1 ): ): 102 - - 115 . . DOI: 10.1109/T-AFFC.2011.28 http://doi.org/10.1109/T-AFFC.2011.28 . .

ME Ayadi , , , MS Kamel , , , F Karray . . Survey on speech emotion recognition: features, classification schemes, and databases . . Patt Recogn , , 2011 . . 44 ( ( 3 ): ): 572 - - 587 . . DOI: 10.1016/j.patcog.2010.09.020 http://doi.org/10.1016/j.patcog.2010.09.020 . .

MM Bradley , , , PJ Lang . . Measuring emotion: the selfassessment manikin and the semantic differential . . J Behav Ther Exp Psych , , 1994 . . 25 ( ( 1 ): ): 49 - - 59 . . DOI: 10.1016/0005-7916(94)90063-9 http://doi.org/10.1016/0005-7916(94)90063-9 . .

C Chandler , , , R Cornes . . Biometric measurement of human emotions . . TECHNIA - Int J Comput Sci Commun Technol , , 2012 . . 4 ( ( 2 ): ): 164 - - 168 . . . .

B Cheng , , , GY Liu . . Emotion recognition from surface EMG signal using wavelet transform and neural network . . J Comput Appl , , 2008 . . 28 ( ( 2 ): ): 333 - - 335 . . . .

NAJ de Witte , , , S Sütterlin , , , C Braet , , , 等 . . Psychophysiological correlates of emotion regulation training in adolescent anxiety: evidence from the novel PIER task . . J Affect Disord , , 2017 . . 214 89 - - 96 . . DOI: 10.1016/j.jad.2017.03.012 http://doi.org/10.1016/j.jad.2017.03.012 . .

ABM Gerdes , , , MJ Wieser , , , GW Alpers . . Emotional pictures and sounds: a review of multimodal interactions of emotion cues in multiple domains . . Front Psychol , , 2014 . . 5 1351 DOI: 10.3389/fpsyg.2014.01351 http://doi.org/10.3389/fpsyg.2014.01351 . .

CD Katsis , , , NS Katertsidis , , , DI Fotiadis . . An integrated system based on physiological signals for the assessment of affective states in patients with anxiety disorders . . Biomed Signal Process Contr , , 2011 . . 6 ( ( 3 ): ): 261 - - 268 . . DOI: 10.1016/j.bspc.2010.12.001 http://doi.org/10.1016/j.bspc.2010.12.001 . .

J Kim , , , E André . . Emotion recognition based on physiological changes in music listening . . IEEE Trans Patt Anal Mach Intell , , 2008 . . 30 ( ( 12 ): ): 2067 - - 2083 . . DOI: 10.1109/TPAMI.2008.26 http://doi.org/10.1109/TPAMI.2008.26 . .

S Koelstra , , , C Muhl , , , M Soleymani , , , 等 . . DEAP: a database for emotion analysis; using physiological signals . . IEEE Trans Affect Comput , , 2012 . . 3 ( ( 1 ): ): 18 - - 31 . . DOI: 10.1109/T-AFFC.2011.15 http://doi.org/10.1109/T-AFFC.2011.15 . .

CL Krumhansl . . An exploratory study of musical emotions and psychophysiology . . Can J Exp Psychol , , 1997 . . 51 ( ( 4 ): ): 336 - - 353 . . DOI: 10.1037/1196-1961.51.4.336 http://doi.org/10.1037/1196-1961.51.4.336 . .

C Li , , , C Xu , , , ZY Feng . . Analysis of physiological for emotion recognition with the IRS model . . Neurocomputing , , 2016 . . 178 103 - - 111 . . DOI: 10.1016/j.neucom.2015.07.112 http://doi.org/10.1016/j.neucom.2015.07.112 . .

SS Li , , , G Walters , , , J Packer , , , 等 . . A comparative analysis of self-report and psychophysiological measures of emotion in the context of tourism advertising . . J Travel Res , , 2017 . . 57 ( ( 8 ): ): 1078 - - 1092 . . DOI: 10.1177/0047287517733555 http://doi.org/10.1177/0047287517733555 . .

Q Lu , , , XO Chen , , , DS Yang , , , 等 . . Boosting for multimodal music emotion classification . . 11 th Int Society for Music Information Retrieval Conf , , 2010 . . p.105 - - 110 . . . .

CLB Maia , , , ES Furtado . . A study about psychophysiological measures in user experience monitoring and evaluation . . Proc 15 th Brazilian Symp on Human Factors in Computing Systems , , 2016 . . p.7 DOI: 10.1145/3033701.3033708 http://doi.org/10.1145/3033701.3033708 . .

MT Mitterschiffthaler , , , CHY Fu , , , JA Dalton , , , 等 . . A functional MRI study of happy and sad affective states induced by classical music . . Hum Brain Mapp , , 2007 . . 28 ( ( 11 ): ): 1150 - - 1162 . . DOI: 10.1002/hbm.20337 http://doi.org/10.1002/hbm.20337 . .

K Mori , , , M Iwanaga . . Two types of peak emotional responses to music: the psychophysiology of chills and tears . . Sci Rep , , 2017 . . 7 46063 DOI: 10.1038/srep46063 http://doi.org/10.1038/srep46063 . .

I Nyklíček , , , JF Thayer , , , LJP van Doornen . . Cardiorespiratory differentiation of musically-induced emotions . . J Psychophysiol , , 1997 . . 11 ( ( 4 ): ): 304 - - 321 . . . .

F Pedregosa , , , G Varoquaux , , , A Gramfort , , , 等 . . Scikitlearn: machine learning in Python . . J Mach Learn Res , , 2011 . . 12 ( ( 10 ): ): 2825 - - 2830 . . . .

RW Picard , , , E Vyzas , , , J Healey . . Toward machine emotional intelligence: analysis of affective physiological state . . IEEE Trans Patt Anal Mach Intell , , 2001 . . 23 ( ( 10 ): ): 1175 - - 1191 . . DOI: 10.1109/34.954607 http://doi.org/10.1109/34.954607 . .

P Rani , , , C Liu , , , N Sarkar , , , 等 . . An empirical study of machine learning techniques for affect recognition in human-robot interaction . . Patt Anal Appl , , 2006 . . 9 ( ( 1 ): ): 58 - - 69 . . DOI: 10.1007/s10044-006-0025-y http://doi.org/10.1007/s10044-006-0025-y . .

P Ren , , , A Barreto , , , Y Gao , , , 等 . . Affective assessment by digital processing of the pupil diameter . . IEEE Trans Affect Comput , , 2013 . . 4 ( ( 1 ): ): 2 - - 14 . . DOI: 10.1109/T-AFFC.2012.25 http://doi.org/10.1109/T-AFFC.2012.25 . .

G Tzanetakis , , , P Cook . . MARSYAS: a framework for audio analysis . . Organ Sound , , 2000 . . 4 ( ( 3 ): ): 169 - - 175 . . DOI: 10.1017/S1355771800003071 http://doi.org/10.1017/S1355771800003071 . .

J Wagner , , , J Kim , , , E Andre . . From physiological signals to emotions: implementing and comparing selected methods for feature extraction and classification . . IEEE Int Conf on Multimedia and Expo , , 2005 . . p.940 - - 943 . . DOI: 10.1109/ICME.2005.1521579 http://doi.org/10.1109/ICME.2005.1521579 . .

SH Wang , , , P Phillips , , , ZC Dong , , , 等 . . Intelligent facial emotion recognition based on stationary wavelet entropy and Jaya algorithm . . Neurocomputing , , 2018 . . 272 668 - - 676 . . DOI: 10.1016/j.neucom.2017.08.015 http://doi.org/10.1016/j.neucom.2017.08.015 . .

WS Wen , , , GY Liu , , , NP Cheng , , , 等 . . Emotion recognition based on multi-variant correlation of physiological signals . . IEEE Trans Affect Comput , , 2014 . . 5 ( ( 2 ): ): 126 - - 140 . . DOI: 10.1109/TAFFC.2014.2327617 http://doi.org/10.1109/TAFFC.2014.2327617 . .

BX Xing , , , KJ Zhang , , , SQ Sun , , , 等 . . Emotion-driven Chinese folk music-image retrieval based on DE-SVM . . Neurocomputing , , 2015 . . 148 619 - - 627 . . DOI: 10.1016/j.neucom.2014.08.007 http://doi.org/10.1016/j.neucom.2014.08.007 . .

YH Yang , , , HH Chen . . Prediction of the distribution of perceived music emotions using discrete samples . . IEEE Trans Audio Speech Lang Process , , 2011 . . 19 ( ( 7 ): ): 2184 - - 2196 . . DOI: 10.1109/TASL.2011.2118752 http://doi.org/10.1109/TASL.2011.2118752 . .

T Yun , , , L Guan . . Human emotional state recognition using real 3D visual features from Gabor library . . Patt Recogn , , 2013 . . 46 ( ( 2 ): ): 529 - - 538 . . DOI: 10.1016/j.patcog.2012.08.002 http://doi.org/10.1016/j.patcog.2012.08.002 . .

YD Zhang , , , ZJ Yang , , , HM Lu , , , 等 . . Facial emotion recognition based on biorthogonal wavelet entropy, fuzzy support vector machine, and stratified cross validation . . IEEE Access , , 2016 . . 4 8375 - - 8385 . . DOI: 10.1109/ACCESS.2016.2628407 http://doi.org/10.1109/ACCESS.2016.2628407 . .

X Zhu . . Emotion recognition of EMG based on BP neural network . . Proc 2 nd Int Symp on Networking and Network Security , , 2010 . . p.227 - - 229 . . . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

A personality-guided affective brain–computer interface for implementation of emotional intelligence in machines

Using Kinect for real-time emotion recognition via facial expressions

Related Author

Shaojie LI

Wei LI

Zejian XING

Wenjie YUAN

Xiangyu WEI

Xiaowei ZHANG

Bin HU

MAO Qi-rong

Related Institution

School of Information Science and Engineering, Lanzhou University

Department of Computer Science and Communication Engineering, Jiangsu University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰