NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images

Luo-yang XUE; Qi-rong MAO; Xiao-hua HUANG; Jie CHEN

doi:10.1631/FITEE.1900618

Your Location：

Home >

Browse articles >

NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images

Regular Papers | Updated：2022-05-19

- NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images
- NLWSNet：基于弱监督学习的嘈杂标签Web图像情感分析
- Frontiers of Information Technology & Electronic Engineering Vol. 21, Issue 9, Pages: 1321-1333(2020)
- Affiliations：
  
  Computer Science and Communication Engineering, Jiangsu University, Zhenjiang 212013, China
  Jiangsu Key Laboratory of Security Technology for Industrial Cyberspace, Zhenjiang 212013, China
  School of Computer Engineering, Nanjing Institute of Technology, Nanjing 211167, China
  Center for Machine Vision and Signal Analysis, University of Oulu, Oulu 8000, Finland
- Author bio：
  
  Qi-rong MAO, E-mail: mao_qr@ujs.edu.cn
- Funds：
- DOI：10.1631/FITEE.1900618
  CLC： TP391.4
- Received：12 November 2019，
  
  Revised：2020-;7-20，
  
  Published：2020-09
- Accepted：
Scan QR Code
Luo-yang XUE, Qi-rong MAO, Xiao-hua HUANG, et al. NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1321-1333.
DOI：

Luo-yang XUE, Qi-rong MAO, Xiao-hua HUANG, et al. NLWSNet: a weakly supervised network for visual sentiment analysis in mislabeled web images[J]. Frontiers of Information Technology & Electronic Engineering, 2020, 21(9): 1321-1333. DOI： 10.1631/FITEE.1900618.

摘要

大规模数据集推动了基于深度卷积神经网络情感分析的快速发展。但是，注释大规模数据集既昂贵又耗时。相反，从网络中很容易获得弱标注的Web图像。当直接使用Web图像训练深度学习模型时，嘈杂标签会导致性能急剧下降。针对这种情况，提出一种端到端的弱监督学习结构，该结构对于弱标签的Web图像具有鲁棒性。具体地说，该注意力模块通过降低训练过程中注意力得分，自动抑制带有错误标签样本的负面影响。另外，在弱监督学习方法中，类激活图模块通过关注正确标签样本的情感区域促进网络学习。除特征学习过程外，将正则化应用于分类器，以最小化同一类别样本的距离，并最大化不同类别样本质心之间的距离。对标记正确和错误的网页图像数据集进行定量和定性评估，结果表明该算法优于现有方法。

Abstract

Large-scale datasets are driving the rapid developments of deep convolutional neural networks for visual sentiment analysis. However

the annotation of large-scale datasets is expensive and time consuming. Instead

it is easy to obtain weakly labeled web images from the Internet. However

noisy labels still lead to seriously degraded performance when we use images directly from the web for training networks. To address this drawback

we propose an end-to-end weakly supervised learning network

which is robust to mislabeled web images. Specifically

the proposed attention module automatically eliminates the distraction of those samples with incorrect labels by reducing their attention scores in the training process. On the other hand

the special-class activation map module is designed to stimulate the network by focusing on the significant regions from the samples with correct labels in a weakly supervised learning approach. Besides the process of feature learning

applying regularization to the classifier is considered to minimize the distance of those samples within the same class and maximize the distance between different class centroids. Quantitative and qualitative evaluations on well- and mislabeled web image datasets demonstrate that the proposed algorithm outperforms the related methods.

关键词

Keywords

references

D Borth , , , RR Ji , , , T Chen , , , 等 . . Large-scale visual sentiment ontology and detectors using adjective noun pairs . . Proc 21 st ACM Int Conf on Multimedia , , 2013 . . p.223 - - 232 . . DOI: 10.1145/2502081.2502282 http://doi.org/10.1145/2502081.2502282 . .

V Campos , , , A Salvador , , , X Giró-i-Nieto , , , 等 . . Diving deep into sentiment: understanding fine-tuned CNNs for visual sentiment prediction , , 2015 . . https://arxiv.org/abs/1508.05056 https://arxiv.org/abs/1508.05056 , , . .

V Campos , , , B Jou , , , X Giró-i-Nieto . . From pixels to sentiment: fine-tuning CNNs for visual sentiment prediction . . Image Vis Comput , , 2017 . . 65 15 - - 22 . . DOI: 10.1016/j.imavis.2017.01.011 http://doi.org/10.1016/j.imavis.2017.01.011 . .

CC Chang , , , CJ Lin . . LIBSVM: a library for support vector machines . . ACM Trans Intell Syst Technol , , 2011 . . 2 ( ( 3 ): ): 27 DOI: 10.1145/1961189.1961199 http://doi.org/10.1145/1961189.1961199 . .

L Chen , , , HW Zhang , , , J Xiao , , , 等 . . SCA-CNN: spatial and channel-wise attention in convolutional networks for image captioning . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.6298 - - 6306 . . DOI: 10.1109/CVPR.2017.667 http://doi.org/10.1109/CVPR.2017.667 . .

SX Chen , , , CJ Zhang , , , M Dong , , , 等 . . Using ranking-CNN for age estimation . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.742 - - 751 . . DOI: 10.1109/CVPR.2017.86 http://doi.org/10.1109/CVPR.2017.86 . .

SX Chen , , , CJ Zhang , , , M Dong . . Coupled end-to-end transfer learning with generalized Fisher information . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018a . . p.4329 - - 4338 . . DOI: 10.1109/CVPR.2018.00455 http://doi.org/10.1109/CVPR.2018.00455 . .

SX Chen , , , CJ Zhang , , , M Dong . . Deep age estimation: from classification to ranking . . IEEE Trans Multim , , 2018b . . 20 ( ( 8 ): ): 2209 - - 2222 . . DOI: 10.1109/TMM.2017.2786869 http://doi.org/10.1109/TMM.2017.2786869 . .

T Chen , , , D Borth , , , T Darrell , , , 等 . . DeepSentiBank: visual sentiment concept classification with deep convolutional neural networks , , 2014a . . https://arxiv.org/abs/1410.8586 https://arxiv.org/abs/1410.8586 , , . .

T Chen , , , FX Yu , , , JW Chen , , , 等 . . Object-based visual sentiment concept analysis and application . . Proc 22 nd ACM Int Conf on Multimedia , , 2014b . . p.367 - - 376 . . DOI: 10.1145/2647868.2654935 http://doi.org/10.1145/2647868.2654935 . .

M Corbetta , , , GL Shulman . . Control of goal-directed and stimulus-driven attention in the brain . . Nat Rev Neurosci , , 2002 . . 3 ( ( 3 ): ): 201 - - 205 . . DOI: 10.1038/nrn755 http://doi.org/10.1038/nrn755 . .

J Deng , , , W Dong , , , R Socher , , , 等 . . ImageNet: a large-scale hierarchical image database . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2009 . . p.248 - - 255 . . DOI: 10.1109/CVPRW.2009.5206848 http://doi.org/10.1109/CVPRW.2009.5206848 . .

T Durand , , , T Mordan , , , N Thome , , , 等 . . WILDCAT: weakly supervised learning of deep ConvNets for image classification, pointwise localization and segmentation . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.5957 - - 5966 . . DOI: 10.1109/CVPR.2017.631 http://doi.org/10.1109/CVPR.2017.631 . .

Y Fang , , , H Tan , , , J Zhang . . Multi-strategy sentiment analysis of consumer reviews based on semantic fuzziness . . IEEE Access , , 2018 . . 6 20625 - - 20631 . . DOI: 10.1109/ACCESS.2018.2820025 http://doi.org/10.1109/ACCESS.2018.2820025 . .

R Girshick , , , J Donahue , , , T Darrell , , , 等 . . Rich feature hierarchies for accurate object detection and semantic segmentation . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2014 . . p.580 - - 587 . . DOI: 10.1109/CVPR.2014.81 http://doi.org/10.1109/CVPR.2014.81 . .

XT He , , , YX Peng . . Weakly supervised learning of part selection model with spatial constraints for fine-grained image classification . . Proc 21 st AAAI Conf on Artificial Intelligence , , 2017 . . p.4075 - - 4081 . . . .

XT He , , , YX Peng , , , JJ Zhao . . Fast fine-grained image classification via weakly supervised discriminative localization . . IEEE Trans Circ Syst Video Technol , , 2019 . . 29 ( ( 5 ): ): 1394 - - 1407 . . DOI: 10.1109/TCSVT.2018.2834480 http://doi.org/10.1109/TCSVT.2018.2834480 . .

GE Hinton . . Visualizing high-dimensional data using t-SNE . . Vigil Christ , , 2008 . . 9 ( ( 2 ): ): 2579 - - 2605 . . . .

J Hu , , , L Shen , , , G Sun . . Squeeze-and-excitation networks . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.7132 - - 7141 . . DOI: 10.1109/CVPR.2018.00745 http://doi.org/10.1109/CVPR.2018.00745 . .

G Huang , , , Z Liu , , , L van der Maaten , , , 等 . . Densely connected convolutional networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.2261 - - 2269 . . DOI: 10.1109/CVPR.2017.243 http://doi.org/10.1109/CVPR.2017.243 . .

L Itti , , , C Koch , , , E Niebur . . A model of saliency-based visual attention for rapid scene analysis . . IEEE Trans Patt Anal Mach Intell , , 1998 . . 20 ( ( 11 ): ): 1254 - - 1259 . . DOI: 10.1109/34.730558 http://doi.org/10.1109/34.730558 . .

XB Jia , , , Y Jin , , , N Li , , , 等 . . Words alignment based on association rules for cross-domain sentiment classification . . Front Inform Technol Electron Eng , , 2018 . . 19 ( ( 2 ): ): 260 - - 272 . . DOI: 10.1631/FITEE.1601679 http://doi.org/10.1631/FITEE.1601679 . .

M Katsurai , , , S Satoh . . Image sentiment analysis using latent correlations among visual, textual, and sentiment views . . Proc IEEE Int Conf on Acoustics, Speech and Signal Processing , , 2016 . . p.2837 - - 2841 . . DOI: 10.1109/ICASSP.2016.7472195 http://doi.org/10.1109/ICASSP.2016.7472195 . .

A Krizhevsky . . Learning Multiple Layers of Features from Tiny Images . . Technical Report TR-2009, University of Toronto, Toronto, Canada , , 2009 . . .

A Krizhevsky , , , I Sutskever , , , GE Hinton . . ImageNet classification with deep convolutional neural networks . . Commun ACM , , 2017 . . 60 ( ( 6 ): ): 84 - - 90 . . DOI: 10.1145/3065386 http://doi.org/10.1145/3065386 . .

Y LeCun , , , B Boser , , , JS Denker , , , 等 . . Backpropagation applied to handwritten zip code recognition . . Neur Comput , , 1989 . . 1 ( ( 4 ): ): 541 - - 551 . . DOI: 10.1162/neco.1989.1.4.541 http://doi.org/10.1162/neco.1989.1.4.541 . .

ZH Li , , , YY Fan , , , WH Liu , , , 等 . . Image sentiment prediction based on textual descriptions with adjective noun pairs . . Multim Tools Appl , , 2018 . . 77 ( ( 1 ): ): 1115 - - 1132 . . DOI: 10.1007/s11042-016-4310-5 http://doi.org/10.1007/s11042-016-4310-5 . .

GL Liu , , , FA Reda , , , KJ Shih , , , 等 . . Image inpainting for irregular holes using partial convolutions . . Proc 15 th European Conf on Computer Vision , , 2018 . . p.89 - - 105 . . DOI: 10.1007/978-3-030-01252-6_6 http://doi.org/10.1007/978-3-030-01252-6_6 . .

J Machajdik , , , A Hanbury . . Affective image classification using features inspired by psychology and art theory . . Proc 18 th ACM Int Conf on Multimedia , , 2010 . . p.83 - - 92 . . DOI: 10.1145/1873951.1873965 http://doi.org/10.1145/1873951.1873965 . .

JA Mikels , , , BL Fredrickson , , , GR Larkin , , , 等 . . Emotional category data on images from the international affective picture system . . Behav Res Methods , , 2005 . . 37 ( ( 4 ): ): 626 - - 630 . . DOI: 10.3758/BF03192732 http://doi.org/10.3758/BF03192732 . .

M Oquab , , , L Bottou , , , I Laptev , , , 等 . . Is object localization for free?—Weakly-supervised learning with convolutional neural networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2015 . . p.685 - - 694 . . DOI: 10.1109/CVPR.2015.7298668 http://doi.org/10.1109/CVPR.2015.7298668 . .

WH Ou , , , X Luan , , , JP Gou , , , 等 . . Robust discriminative nonnegative dictionary learning for occluded face recognition . . Patt Recogn Lett , , 2018 . . 107 41 - - 49 . . DOI: 10.1016/j.patrec.2017.07.006 http://doi.org/10.1016/j.patrec.2017.07.006 . .

J Park , , , S Woo , , , JY Lee , , , 等 . . BAM: bottleneck attention module . . Proc British Machine Vision Conf, Article 147 , , 2018 . . .

KC Peng , , , A Sadovnik , , , A Gallagher , , , 等 . . Where do emotions come from . . Predicting the emotion stimuli map. Proc IEEE Int Conf on Image Processing , , 2016 . . p.614 - - 618 . . DOI: 10.1109/ICIP.2016.7532430 http://doi.org/10.1109/ICIP.2016.7532430 . .

YX Peng , , , JW Qi , , , YK Zhuo . . MAVA: multi-level adaptive visual-textual alignment by cross-media bi-attention mechanism . . IEEE Trans Image Process , , 2019a . . 29 2728 - - 2741 . . DOI: 10.1109/TIP.2019.2952085 http://doi.org/10.1109/TIP.2019.2952085 . .

YX Peng , , , YZ Zhao , , , JC Zhang . . Two-stream collaborative learning with spatial-temporal attention for video classification . . IEEE Trans Circ Syst Video Technol , , 2019b . . 29 ( ( 3 ): ): 773 - - 786 . . DOI: 10.1109/TCSVT.2018.2808685 http://doi.org/10.1109/TCSVT.2018.2808685 . .

A Rohrbach , , , M Rohrbach , , , RH Hu , , , 等 . . Grounding of textual phrases in images by reconstruction . . Proc 14 th European Conf on Computer Vision , , 2016 . . p.817 - - 834 . . DOI: 10.1007/978-3-319-46448-0_49 http://doi.org/10.1007/978-3-319-46448-0_49 . .

K Simonyan , , , A Zisserman . . Very deep convolutional networks for large-scale image recognition , , 2014 . . https://arxiv.org/abs/1409.1556 https://arxiv.org/abs/1409.1556 , , . .

M Sun , , , JF Yang , , , K Wang , , , 等 . . Discovering affective regions in deep convolutional neural networks for visual sentiment prediction . . Proc IEEE Int Conf on Multimedia and Expo , , 2016 . . p.1 - - 6 . . DOI: 10.1109/ICME.2016.7552961 http://doi.org/10.1109/ICME.2016.7552961 . .

C Szegedy , , , V Vanhoucke , , , S Ioffe , , , 等 . . Rethinking the inception architecture for computer vision . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.2818 - - 2826 . . DOI: 10.1109/CVPR.2016.308 http://doi.org/10.1109/CVPR.2016.308 . .

F Wang , , , MQ Jiang , , , C Qian , , , 等 . . Residual attention network for image classification . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.6450 - - 6458 . . DOI: 10.1109/CVPR.2017.683 http://doi.org/10.1109/CVPR.2017.683 . .

S Woo , , , J Park , , , JY Lee , , , 等 . . CBAM: convolutional block attention module . . Proc 15 th European Conf on Computer Vision , , 2018 . . p.3 - - 19 . . DOI: 10.1007/978-3-030-01234-2_1 http://doi.org/10.1007/978-3-030-01234-2_1 . .

FY Xiao , , , L Sigal , , , YJ Lee . . Weakly-supervised visual grounding of phrases with linguistic structures . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.5253 - - 5262 . . DOI: 10.1109/CVPR.2017.558 http://doi.org/10.1109/CVPR.2017.558 . .

JF Yang , , , DY She , , , M Sun . . Joint image emotion classification and distribution learning via deep convolutional neural network . . Proc 26 th Int Joint Conf on Artificial Intelligence , , 2017a . . p.3266 - - 3272 . . DOI: 10.24963/ijcai.2017/456 http://doi.org/10.24963/ijcai.2017/456 . .

JF Yang , , , M Sun , , , XX Sun . . Learning visual sentiment distributions via augmented conditional probability neural network . . Proc 31 st AAAI Conf on Artificial Intelligence , , 2017b . . p.224 - - 230 . . . .

JF Yang , , , DY She , , , M Sun , , , 等 . . Visual sentiment prediction based on automatic discovery of affective regions . . IEEE Trans Multim , , 2018a . . 20 ( ( 9 ): ): 2513 - - 2525 . . DOI: 10.1109/TMM.2018.2803520 http://doi.org/10.1109/TMM.2018.2803520 . .

JF Yang , , , DY She , , , YK Lai , , , 等 . . Weakly supervised coupled networks for visual sentiment analysis . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018b . . p.7584 - - 7592 . . DOI: 10.1109/CVPR.2018.00791 http://doi.org/10.1109/CVPR.2018.00791 . .

QZ You , , , JB Luo , , , HL Jin , , , 等 . . Robust image sentiment analysis using progressively trained and domain transferred deep networks . . Proc 29 th AAAI Conf on Artificial Intelligence , , 2015 . . p.381 - - 388 . . . .

QZ You , , , JB Luo , , , HL Jin , , , 等 . . Building a large scale dataset for image emotion recognition: the fine print and the benchmark . . Proc 30 th AAAI Conf on Artificial Intelligence , , 2016 . . p.308 - - 314 . . . .

QZ You , , , HL Jin , , , JB Luo . . Visual sentiment analysis by attending on local image regions . . Proc 31 st AAAI Conf on Artificial Intelligence , , 2017 . . p.231 - - 237 . . . .

JH Yu , , , Z Lin , , , JM Yang , , , 等 . . Generative image inpainting with contextual attention . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018 . . p.5505 - - 5514 . . DOI: 10.1109/CVPR.2018.00577 http://doi.org/10.1109/CVPR.2018.00577 . .

JB Yuan , , , S Mcdonough , , , QZ You , , , 等 . . Sentribute: image sentiment analysis from a mid-level perspective . . Proc 2 nd Int Workshop on Issues of Sentiment Discovery and Opinion Mining , , 2013 . . p.1 - - 8 . . DOI: 10.1145/2502069.2502079 http://doi.org/10.1145/2502069.2502079 . .

S Zagoruyko , , , N Komodakis . . Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer , , 2017 . . https://arxiv.org/abs/1612.03928 https://arxiv.org/abs/1612.03928 , , . .

SN Zeng , , , JP Gou , , , X Yang . . Improving sparsity of coefficients for robust sparse and collaborative representation-based image classification . . Neur Comput Appl , , 2018 . . 30 ( ( 10 ): ): 2965 - - 2978 . . DOI: 10.1007/s00521-017-2900-4 http://doi.org/10.1007/s00521-017-2900-4 . .

FF Zhang , , , QR Mao , , , XJ Shen , , , 等 . . Spatially coherent feature learning for pose-invariant facial expression recognition . . ACM Trans Multim Comput Commun Appl , , 2018a . . 14 ( ( 1s ): ): 27 DOI: 10.1145/3176646 http://doi.org/10.1145/3176646 . .

FF Zhang , , , TZ Zhang , , , QR Mao , , , 等 . . Joint pose and expression modeling for facial expression recognition . . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , , 2018b . . p.3359 - - 3368 . . DOI: 10.1109/CVPR.2018.00354 http://doi.org/10.1109/CVPR.2018.00354 . .

N Zhang , , , J Donahue , , , R Girshick , , , 等 . . Part-based R-CNNs for fine-grained category detection . . Proc 13 th European Conf on Computer Vision , , 2014 . . p.834 - - 849 . . DOI: 10.1007/978-3-319-10590-1_54 http://doi.org/10.1007/978-3-319-10590-1_54 . .

QS Zhang , , , SC Zhu . . Visual interpretability for deep learning: a survey . . Front Inform Technol Electron Eng , , 2018 . . 19 ( ( 1 ): ): 27 - - 39 . . DOI: 10.1631/FITEE.1700808 http://doi.org/10.1631/FITEE.1700808 . .

SC Zhao , , , Y Gao , , , XL Jiang , , , 等 . . Exploring principles-of-art features for image emotion recognition . . Proc 22 nd ACM Int Conf on Multimedia , , 2014 . . p.47 - - 56 . . DOI: 10.1145/2647868.2654930 http://doi.org/10.1145/2647868.2654930 . .

SC Zhao , , , HX Yao , , , Y Gao , , , 等 . . Predicting personalized emotion perceptions of social images . . Proc 24 th ACM Int Conf on Multimedia , , 2016 . . p.1385 - - 1394 . . DOI: 10.1145/2964284.2964289 http://doi.org/10.1145/2964284.2964289 . .

SC Zhao , , , GG Ding , , , Y Gao , , , 等 . . Approximating discrete probability distribution of image emotions by multi-modal features fusion . . Proc 26 th Int Joint Conf on Artificial Intelligence , , 2017 . . p.4669 - - 4675 . . DOI: 10.24963/ijcai.2017/651 http://doi.org/10.24963/ijcai.2017/651 . .

BL Zhou , , , A Khosla , , , A Lapedriza , , , 等 . . Learning deep features for discriminative localization . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.2921 - - 2929 . . DOI: 10.1109/CVPR.2016.319 http://doi.org/10.1109/CVPR.2016.319 . .

Y Zhu , , , YZ Zhou , , , QX Ye , , , 等 . . Soft proposal networks for weakly supervised object localization . . Proc IEEE Int Conf on Computer Vision , , 2017 . . p.1859 - - 1868 . . DOI: 10.1109/ICCV.2017.204 http://doi.org/10.1109/ICCV.2017.204 . .

YK Zhu , , , O Groth , , , M Bernstein , , , 等 . . Visual7W: grounded question answering in images . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.4995 - - 5004 . . DOI: 10.1109/CVPR.2016.540 http://doi.org/10.1109/CVPR.2016.540 . .

BH Zhuang , , , LQ Liu , , , Y Li , , , 等 . . Attend in groups: a weakly-supervised deep learning framework for learning from web data . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.2915 - - 2924 . . DOI: 10.1109/CVPR.2017.311 http://doi.org/10.1109/CVPR.2017.311 . .

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

No data

Related Author

No data

Related Institution

No data

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰