Visual interpretability for deep learning: a survey

Quan-shi ZHANG; Song-chun ZHU

doi:10.1631/FITEE.1700808

Your Location：

Home >

Browse articles >

Visual interpretability for deep learning: a survey

2018 Special Issue on Artificial Intelligence 2.0: Theories and Applications | Updated：2022-05-19

- Visual interpretability for deep learning: a survey
- 深度学习中的视觉可解释性
- Frontiers of Information Technology & Electronic Engineering Vol. 19, Issue 1, Pages: 27-39(2018)
- Affiliations：
  
  University of California, Los Angeles, California 90095, USA
- Author bio：
  
  Quan-shi ZHANG, E-mail: zhangqs@ucla.edu
  [ "Song-chun ZHU,E-mail: sczhu@stat.ucla.edu" ]
- Funds：
- DOI：10.1631/FITEE.1700808
  CLC： TP391
- Received：02 December 2017，
  
  Revised：2018-;1-28，
  
  Published：2018-01
- Accepted：
Scan QR Code
Quan-shi ZHANG, Song-chun ZHU. Visual interpretability for deep learning: a survey[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(1): 27-39.
DOI：

Quan-shi ZHANG, Song-chun ZHU. Visual interpretability for deep learning: a survey[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(1): 27-39. DOI： 10.1631/FITEE.1700808.

摘要

总结了近年来在理解神经网络内部特征表达和训练一个具有中层表达可解释性的深度神经网络上的相关研究工作。虽然深度神经网络在众多人工智能任务中已有杰出表现，但神经网络中层表达的可解释性依然是该领域发展的重大瓶颈。目前，深度神经网络以低解释性的黑箱表达为代价，获取了强大的分类能力。我们认为提高神经网络中层特征表达的可解释性，可以帮助人们打破众多深度学习的发展瓶颈，比如，小数据训练，语义层面上的人机交互式训练，以及基于内在特征语义定向精准修复网络中层特征表达缺陷等难题。本文着眼于卷积神经网络，调研了：(1) 网络表达可视化方法；(2) 网络表达的诊断方法；(3) 自动解构解释卷积神经网络的方法；(4) 学习中层特征表达可解释的神经网络的方法；(5) 基于网络可解释性的中层对端的深度学习算法。最后，讨论了可解释性人工智能未来可能的发展趋势。

Abstract

This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks

interpretability is always Achilles' heel of deep neural networks. At present

deep neural networks obtain high discrimination power at the cost of a low interpretability of their black-box representations. We believe that high model interpretability may help people break several bottlenecks of deep learning

e.g.

learning from a few annotations

learning via human-computer communications at the semantic level

and semantically debugging network representations. We focus on convolutional neural networks (CNNs)

and revisit the visualization of CNN representations

methods of diagnosing representations of pre-trained CNNs

approaches for disentangling pre-trained CNN representations

learning of CNNs with disentangled representations

and middle-to-end learning based on model interpretability. Finally

we discuss prospective trends in explainable artificial intelligence.

关键词

Keywords

references

M Aubry , , , BC Russell . . Understanding deep features with computer-generated imagery . . IEEE Int Conf on Computer Vision , , 2015 . . p.2875 - - 2883 . . DOI: 10.1109/ICCV.2015.329 http://doi.org/10.1109/ICCV.2015.329 . .

M Aubry , , , D Maturana , , , A Efros , , , 等 . . Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2014 . . p.3762 - - 3769 . . . .

D Bau , , , B Zhou , , , A Khosla , , , 等 . . Network dissection: quantifying interpretability of deep visual representations . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.1063 - - 6919 . . DOI: 10.1109/CVPR.2017.354 http://doi.org/10.1109/CVPR.2017.354 . .

X Chen , , , Y Duan , , , R Houthooft , , , 等 . . Infogan: interpretable representation learning by information maximizing generative adversarial nets . . NIPS , , 2016 . . p.2172 - - 2180 . . . .

A Dosovitskiy , , , T Brox . . Inverting visual representations with convolutional networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.4829 - - 4837 . . DOI: 10.1109/CVPR.2016.522 http://doi.org/10.1109/CVPR.2016.522 . .

RC Fong , , , A Vedaldi . . Interpretable explanations of black boxes by meaningful perturbation . . IEEE Int Conf on Computer Vision , , 2017 . . p.3429 - - 3437 . . DOI: 10.1109/ICCV.2017.371 http://doi.org/10.1109/ICCV.2017.371 . .

Y Goyal , , , A Mohapatra , , , D Parikh , , , 等 . . Towards transparent AI systems: interpreting visual question answering models . . https://arxiv.org/abs/1608.08974 , , 2016 . . .

K He , , , X Zhang , , , S Ren , , , 等 . . Deep residual learning for image recognition . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.770 - - 778 . . DOI: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90 . .

Z Hu , , , X Ma , , , Z Liu , , , 等 . . Harnessing deep neural networks with logic rules . . http://arxiv.org/abs/1603.06318 , , 2016 . . .

G Huang , , , Z Liu , , , KQ Weinberger , , , 等 . . Densely connected convolutional networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.4700 - - 4708 . . . .

PJ Kindermans , , , KT Schtt , , , M Alber , , , 等 . . Learning how to explain neural networks: patternnet and patternattribution . . http://arxiv.org/abs/1705.05598 , , 2017 . . .

P Koh , , , P Liang . . Understanding black-box predictions via influence functions . . Proc 34 th Int Conf on Machine Learning , , 2017 . . p.1885 - - 1894 . . . .

A Krizhevsky , , , I Sutskever , , , GE Hinton . . Imagenet classification with deep convolutional neural networks . . NIPS , , 2012 . . p.1097 - - 1105 . . . .

D Kumar , , , A Wong , , , GW Taylor . . Explaining the unexplained: a class-enhanced attentive response (clear) approach to understanding deep neural networks . . IEEE Conf on Computer Vision and Pattern Recognition Workshops , , 2017 . . p.1686 - - 1694 . . DOI: 10.1109/CVPRW.2017.215 http://doi.org/10.1109/CVPRW.2017.215 . .

H Lakkaraju , , , E Kamar , , , R Caruana , , , 等 . . Identifying unknown unknowns in the open world: representations and policies for guided exploration . . Proc 31 st AAAI Conf on Artificial Intelligence , , 2017 . . p.2124 - - 2132 . . . .

Y LeCun , , , L Bottou , , , Y Bengio , , , 等 . . Gradient-based learning applied to document recognition . . Proc IEEE , , 1998a . . 86 ( ( 11 ): ): 2278 - - 2324 . . DOI: 10.1109/5.726791 http://doi.org/10.1109/5.726791 DOI: 10.1109/5.726791 http://doi.org/10.1109/5.726791 . .

Y LeCun , , , C Cortes , , , CJ Burges . . The MNIST Database of Handwritten Digits . . http://yann.lecun.com/exdb/mnist/ [Accessed on June, 2017] , , 1998b . . 2017 . .

Z Liu , , , P Luo , , , X Wang , , , 等 . . Deep learning face attributes in the wild . . IEEE Int Conf on Computer Vision , , 2015 . . p.3730 - - 3738 . . DOI: 10.1109/ICCV.2015.425 http://doi.org/10.1109/ICCV.2015.425 . .

Y Lu . . Unsupervised learning on neural network outputs (v9) . . http://arxiv.org/abs/1506.00990 , , 2015 . . .

A Mahendran , , , A Vedaldi . . Understanding deep image representations by inverting them . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2015 . . p.5188 - - 5196 . . DOI: 10.1109/CVPR.2015.7299155 http://doi.org/10.1109/CVPR.2015.7299155 . .

Y Netzer , , , T Wang , , , A Coates , , , 等 . . Reading digits in natural images with unsupervised feature learning . . NIPS , , 2011 . . p.1 - - 9 . . . .

A Nguyen , , , J Clune , , , Y Bengio , , , 等 . . Plug & play generative networks: conditional iterative generation of images in latent space . . IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.3510 - - 3520 . . DOI: 10.1109/CVPR.2017.374 http://doi.org/10.1109/CVPR.2017.374 . .

C Olah , , , A Mordvintsev , , , L Schubert . . Feature visualization . . Distill , , 2017 . . DOI: 10.23915/distill.00007 http://doi.org/10.23915/distill.00007 . .

P Paysan , , , R Knothe , , , B Amberg , , , 等 . . A 3D face model for pose and illumination invariant face recognition . . 6 th IEEE Int Conf on Advanced Video and Signal Based Surveillance , , 2009 . . p.296 - - 301 . . DOI: 10.1109/AVSS.2009.58 http://doi.org/10.1109/AVSS.2009.58 . .

MT Ribeiro , , , S Singh , , , C Guestrin . . "Why should I trust you . . " explaining the predictions of any classifier. Proc 22 nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining , , 2016 . . p.1135 - - 1144 . . DOI: 10.1145/2939672.2939778 http://doi.org/10.1145/2939672.2939778 . .

S Sabour , , , N Frosst , , , GE Hinton . . Dynamic routing between capsules . . NIPS , , 2017 . . p.3859 - - 3869 . . . .

RR Selvaraju , , , M Cogswell , , , A Das , , , 等 . . Grad-CAM: visual explanations from deep networks via gradient-based localization . . IEEE Int Conf on Computer Vision , , 2017 . . p.618 - - 626 . . DOI: 10.1109/ICCV.2017.74 http://doi.org/10.1109/ICCV.2017.74 . .

K Simonyan , , , A Vedaldi , , , A Zisserman . . Deep inside convolutional networks: visualising image classification models and saliency maps . . http://arxiv.org/abs/1312.6034 , , 2013 . . .

JT Springenberg , , , A Dosovitskiy , , , T Brox , , , 等 . . Striving for simplicity: the all convolutional net . . Inte Conf on Learning Representations , , 2015 . . p.1 - - 14 . . . .

J Su , , , DV Vargas , , , S Kouichi . . One pixel attack for fooling deep neural networks . . http://arxiv.org/abs/1710.08864 , , 2017 . . .

C Szegedy , , , W Zaremba , , , I Sutskever , , , 等 . . Intriguing properties of neural networks . . http://arxiv.org/abs/1312.6199 , , 2014 . . .

P Wang , , , Q Wu , , , C Shen , , , 等 . . The VQA-machine: learning how to use existing vision algorithms to answer new questions . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017 . . p.1173 - - 1182 . . DOI: 10.1109/CVPR.2017.416 http://doi.org/10.1109/CVPR.2017.416 . .

TF Wu , , , SC Zhu . . A numerical study of the bottom-up and top-down inference processes in And-Or graphs . . Int J Comput Vis , , 2011 . . 93 ( ( 2 ): ): 226 - - 252 . . . .

TF Wu , , , GS Xia , , , SC Zhu . . Compositional boosting for computing hierarchical image structures . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2007 . . p.1 - - 8 . . DOI: 10.1109/CVPR.2007.383034 http://doi.org/10.1109/CVPR.2007.383034 . .

TF Wu , , , X Li , , , X Song , , , 等 . . Interpretable R-CNN . . http://arxiv.org/abs/1711.05226 , , 2017 . . .

X Yang , , , TF Wu , , , SC Zhu . . Evaluating information contributions of bottom-up and top-down processes . . IEEE 12 th Int Conf on Computer Vision , , 2009 . . p.1042 - - 1049 . . DOI: 10.1109/ICCV.2009.5459386 http://doi.org/10.1109/ICCV.2009.5459386 . .

J Yosinski , , , J Clune , , , Y Bengio , , , 等 . . How transferable are features in deep neural networks . . NIPS , , 2014 . . p.1173 - - 1182 . . . .

MD Zeiler , , , R Fergus . . Visualizing and understanding convolutional networks . . European Conf on Computer Vision , , 2014 . . p.818 - - 833 . . DOI: 10.1007/978-3-319-10590-1_53 http://doi.org/10.1007/978-3-319-10590-1_53 . .

Q Zhang , , , R Cao , , , YN Wu , , , 等 . . Growing interpretable part graphs on convnets via multi-shot learning . . Proc 30 th AAAI Conf on Artificial Intelligence , , 2016 . . p.2898 - - 2906 . . . .

Q Zhang , , , R Cao , , , YN Wu , , , 等 . . Mining object parts from CNNs via active question-answering . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2017a . . p.346 - - 355 . . DOI: 10.1109/CVPR.2017.414 http://doi.org/10.1109/CVPR.2017.414 . .

Q Zhang , , , R Cao , , , S Zhang , , , 等 . . Interactively transferring CNN patterns for part localization . . http://arxiv.org/abs/1708.01783 , , 2017b . . .

Q Zhang , , , W Wang , , , SC Zhu . . Examining CNN representations with respect to dataset bias . . Proc 32 nd AAAI Conf on Artificial Intelligence, in press , , 2018a . . .

Q Zhang , , , R Cao , , , F Shi , , , 等 . . Interpreting CNN knowledge via an explanatory graph . . Proc 32 nd AAAI Conf on Artificial Intelligence , , 2018b . . p.2124 - - 2132 . . . .

Q Zhang , , , Y Yang , , , YN Wu , , , 等 . . Interpreting CNNs via decision trees . . http://arxiv.org/abs/1802.00121 , , 2018c . . .

Q Zhang , , , YN Wu , , , SC Zhu . . Interpretable convolutional neural networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition, in press , , 2018d . . .

B Zhou , , , A Khosla , , , A Lapedriza , , , 等 . . Object detectors emerge in deep scene CNNs . . http://arxiv.org/abs/1412.6856 , , 2015 . . .

LM Zintgraf , , , TSCT Adel , , , M Welling . . Visualizing deep neural network decisions: prediction difference analysis . . http://arxiv.org/abs/1702.04595 , , 2017 . . .

Views

110

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Visual knowledge in the big model era: retrospect and prospect

Quant 4.0: engineering quantitative investment with automated, explainable, and knowledge-driven artificial intelligence

Accurate estimation of 6-DoF tooth pose in 3D intraoral scans for dental applications using deep learning

Smart grid dispatch powered by deep learning: a survey

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Related Author

Wenguan WANG

Yi YANG

Yunhe PAN

Jian GUO

Saizhuo WANG

Lionel M. NI

Heung-Yeung SHUM

Wanghui DING

Related Institution

College of Computer Science and Technology, Zhejiang University

IDEA Research, International Digital Economy Academy

The Hong Kong University of Science and Technology

Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Zhejiang Provincial Clinical Research Center for Oral Diseases, Key Laboratory of Oral Biomedical Research of Zhejiang Province, Cancer Center of Zhejiang University, Engineering Research Center of Oral Biomaterials and Devices of Zhejiang Province

Zhejiang University–University of Illinois at Urbana-Champaign Institute, Zhejiang University

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰