Visual interpretability for deep learning: a survey

Quan-shi ZHANG; Song-chun ZHU

doi:10.1631/FITEE.1700808

Your Location：

Home >

Browse articles >

Visual interpretability for deep learning: a survey

2018 Special Issue on Artificial Intelligence 2.0: Theories and Applications | Updated：2022-05-19

- Visual interpretability for deep learning: a survey
  Enhanced Publication
- 深度学习中的视觉可解释性
- Frontiers of Information Technology & Electronic Engineering Vol. 19, Issue 1, Pages: 27-39(2018)
- Affiliations：
  
  University of California, Los Angeles, California 90095, USA
- Author bio：
  
  Quan-shi ZHANG, E-mail: zhangqs@ucla.edu
  [ "Song-chun ZHU,E-mail: sczhu@stat.ucla.edu" ]
- Funds：
- DOI：10.1631/FITEE.1700808
  CLC： TP391
- Published：2018-01，
  
  Received：02 December 2017，
  
  Revised：28 January 2018，
- Accepted：
Scan QR Code
QUAN-SHI ZHANG, SONG-CHUN ZHU. Visual interpretability for deep learning: a survey. [J]. Frontiers of information technology & electronic engineering, 2018, 19(1): 27-39.
DOI：

QUAN-SHI ZHANG, SONG-CHUN ZHU. Visual interpretability for deep learning: a survey. [J]. Frontiers of information technology & electronic engineering, 2018, 19(1): 27-39. DOI： 10.1631/FITEE.1700808.

摘要

总结了近年来在理解神经网络内部特征表达和训练一个具有中层表达可解释性的深度神经网络上的相关研究工作。虽然深度神经网络在众多人工智能任务中已有杰出表现，但神经网络中层表达的可解释性依然是该领域发展的重大瓶颈。目前，深度神经网络以低解释性的黑箱表达为代价，获取了强大的分类能力。我们认为提高神经网络中层特征表达的可解释性，可以帮助人们打破众多深度学习的发展瓶颈，比如，小数据训练，语义层面上的人机交互式训练，以及基于内在特征语义定向精准修复网络中层特征表达缺陷等难题。本文着眼于卷积神经网络，调研了：(1) 网络表达可视化方法；(2) 网络表达的诊断方法；(3) 自动解构解释卷积神经网络的方法；(4) 学习中层特征表达可解释的神经网络的方法；(5) 基于网络可解释性的中层对端的深度学习算法。最后，讨论了可解释性人工智能未来可能的发展趋势。

Abstract

This paper reviews recent studies in understanding neural-network representations and learning neural networks with interpretable/disentangled middle-layer representations. Although deep neural networks have exhibited superior performance in various tasks

interpretability is always Achilles' heel of deep neural networks. At present

deep neural networks obtain high discrimination power at the cost of a low interpretability of their black-box representations. We believe that high model interpretability may help people break several bottlenecks of deep learning

e.g.

learning from a few annotations

learning via human-computer communications at the semantic level

and semantically debugging network representations. We focus on convolutional neural networks (CNNs)

and revisit the visualization of CNN representations

methods of diagnosing representations of pre-trained CNNs

approaches for disentangling pre-trained CNN representations

learning of CNNs with disentangled representations

and middle-to-end learning based on model interpretability. Finally

we discuss prospective trends in explainable artificial intelligence.

关键词

人工智能深度学习可解释性模型

Keywords

Artificial intelligenceDeep learningInterpretable model

references

M Aubry, , , BC Russell. . Understanding deep features with computer-generated imagery. . IEEE Int Conf on Computer Vision, , 2015. . p.2875--2883. . DOI:10.1109/ICCV.2015.329http://doi.org/10.1109/ICCV.2015.329..

M Aubry, , , D Maturana, , , A Efros, , , 等. . Seeing 3D chairs: exemplar part-based 2D-3D alignment using a large dataset of CAD models. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2014. . p.3762--3769. . ..

D Bau, , , B Zhou, , , A Khosla, , , 等. . Network dissection: quantifying interpretability of deep visual representations. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2017. . p.1063--6919. . DOI:10.1109/CVPR.2017.354http://doi.org/10.1109/CVPR.2017.354..

X Chen, , , Y Duan, , , R Houthooft, , , 等. . Infogan: interpretable representation learning by information maximizing generative adversarial nets. . NIPS, , 2016. . p.2172--2180. . ..

A Dosovitskiy, , , T Brox. . Inverting visual representations with convolutional networks. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2016. . p.4829--4837. . DOI:10.1109/CVPR.2016.522http://doi.org/10.1109/CVPR.2016.522..

RC Fong, , , A Vedaldi. . Interpretable explanations of black boxes by meaningful perturbation. . IEEE Int Conf on Computer Vision, , 2017. . p.3429--3437. . DOI:10.1109/ICCV.2017.371http://doi.org/10.1109/ICCV.2017.371..

Y Goyal, , , A Mohapatra, , , D Parikh, , , 等. . Towards transparent AI systems: interpreting visual question answering models. . https://arxiv.org/abs/1608.08974, , 2016. ..

K He, , , X Zhang, , , S Ren, , , 等. . Deep residual learning for image recognition. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2016. . p.770--778. . DOI:10.1109/CVPR.2016.90http://doi.org/10.1109/CVPR.2016.90..

Z Hu, , , X Ma, , , Z Liu, , , 等. . Harnessing deep neural networks with logic rules. . http://arxiv.org/abs/1603.06318, , 2016. ..

G Huang, , , Z Liu, , , KQ Weinberger, , , 等. . Densely connected convolutional networks. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2017. . p.4700--4708. . ..

PJ Kindermans, , , KT Schtt, , , M Alber, , , 等. . Learning how to explain neural networks: patternnet and patternattribution. . http://arxiv.org/abs/1705.05598, , 2017. ..

P Koh, , , P Liang. . Understanding black-box predictions via influence functions. . Proc 34th Int Conf on Machine Learning, , 2017. . p.1885--1894. . ..

A Krizhevsky, , , I Sutskever, , , GE Hinton. . Imagenet classification with deep convolutional neural networks. . NIPS, , 2012. . p.1097--1105. . ..

D Kumar, , , A Wong, , , GW Taylor. . Explaining the unexplained: a class-enhanced attentive response (clear) approach to understanding deep neural networks. . IEEE Conf on Computer Vision and Pattern Recognition Workshops, , 2017. . p.1686--1694. . DOI:10.1109/CVPRW.2017.215http://doi.org/10.1109/CVPRW.2017.215..

H Lakkaraju, , , E Kamar, , , R Caruana, , , 等. . Identifying unknown unknowns in the open world: representations and policies for guided exploration. . Proc 31st AAAI Conf on Artificial Intelligence, , 2017. . p.2124--2132. . ..

Y LeCun, , , L Bottou, , , Y Bengio, , , 等. . Gradient-based learning applied to document recognition. . Proc IEEE, , 1998a. . 86((11):):2278--2324. . DOI:10.1109/5.726791http://doi.org/10.1109/5.726791DOI:10.1109/5.726791http://doi.org/10.1109/5.726791..

Y LeCun, , , C Cortes, , , CJ Burges. . The MNIST Database of Handwritten Digits. . http://yann.lecun.com/exdb/mnist/ [Accessed on June, 2017], , 1998b. . 2017..

Z Liu, , , P Luo, , , X Wang, , , 等. . Deep learning face attributes in the wild. . IEEE Int Conf on Computer Vision, , 2015. . p.3730--3738. . DOI:10.1109/ICCV.2015.425http://doi.org/10.1109/ICCV.2015.425..

Y Lu. . Unsupervised learning on neural network outputs (v9). . http://arxiv.org/abs/1506.00990, , 2015. ..

A Mahendran, , , A Vedaldi. . Understanding deep image representations by inverting them. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2015. . p.5188--5196. . DOI:10.1109/CVPR.2015.7299155http://doi.org/10.1109/CVPR.2015.7299155..

Y Netzer, , , T Wang, , , A Coates, , , 等. . Reading digits in natural images with unsupervised feature learning. . NIPS, , 2011. . p.1--9. . ..

A Nguyen, , , J Clune, , , Y Bengio, , , 等. . Plug & play generative networks: conditional iterative generation of images in latent space. . IEEE Conf on Computer Vision and Pattern Recognition, , 2017. . p.3510--3520. . DOI:10.1109/CVPR.2017.374http://doi.org/10.1109/CVPR.2017.374..

C Olah, , , A Mordvintsev, , , L Schubert. . Feature visualization. . Distill, , 2017. . DOI:10.23915/distill.00007http://doi.org/10.23915/distill.00007..

P Paysan, , , R Knothe, , , B Amberg, , , 等. . A 3D face model for pose and illumination invariant face recognition. . 6th IEEE Int Conf on Advanced Video and Signal Based Surveillance, , 2009. . p.296--301. . DOI:10.1109/AVSS.2009.58http://doi.org/10.1109/AVSS.2009.58..

MT Ribeiro, , , S Singh, , , C Guestrin. . "Why should I trust you. . " explaining the predictions of any classifier. Proc 22nd ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining, , 2016. . p.1135--1144. . DOI:10.1145/2939672.2939778http://doi.org/10.1145/2939672.2939778..

S Sabour, , , N Frosst, , , GE Hinton. . Dynamic routing between capsules. . NIPS, , 2017. . p.3859--3869. . ..

RR Selvaraju, , , M Cogswell, , , A Das, , , 等. . Grad-CAM: visual explanations from deep networks via gradient-based localization. . IEEE Int Conf on Computer Vision, , 2017. . p.618--626. . DOI:10.1109/ICCV.2017.74http://doi.org/10.1109/ICCV.2017.74..

K Simonyan, , , A Vedaldi, , , A Zisserman. . Deep inside convolutional networks: visualising image classification models and saliency maps. . http://arxiv.org/abs/1312.6034, , 2013. ..

JT Springenberg, , , A Dosovitskiy, , , T Brox, , , 等. . Striving for simplicity: the all convolutional net. . Inte Conf on Learning Representations, , 2015. . p.1--14. . ..

J Su, , , DV Vargas, , , S Kouichi. . One pixel attack for fooling deep neural networks. . http://arxiv.org/abs/1710.08864, , 2017. ..

C Szegedy, , , W Zaremba, , , I Sutskever, , , 等. . Intriguing properties of neural networks. . http://arxiv.org/abs/1312.6199, , 2014. ..

P Wang, , , Q Wu, , , C Shen, , , 等. . The VQA-machine: learning how to use existing vision algorithms to answer new questions. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2017. . p.1173--1182. . DOI:10.1109/CVPR.2017.416http://doi.org/10.1109/CVPR.2017.416..

TF Wu, , , SC Zhu. . A numerical study of the bottom-up and top-down inference processes in And-Or graphs. . Int J Comput Vis, , 2011. . 93((2):):226--252. . ..

TF Wu, , , GS Xia, , , SC Zhu. . Compositional boosting for computing hierarchical image structures. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2007. . p.1--8. . DOI:10.1109/CVPR.2007.383034http://doi.org/10.1109/CVPR.2007.383034..

TF Wu, , , X Li, , , X Song, , , 等. . Interpretable R-CNN. . http://arxiv.org/abs/1711.05226, , 2017. ..

X Yang, , , TF Wu, , , SC Zhu. . Evaluating information contributions of bottom-up and top-down processes. . IEEE 12th Int Conf on Computer Vision, , 2009. . p.1042--1049. . DOI:10.1109/ICCV.2009.5459386http://doi.org/10.1109/ICCV.2009.5459386..

J Yosinski, , , J Clune, , , Y Bengio, , , 等. . How transferable are features in deep neural networks. . NIPS, , 2014. . p.1173--1182. . ..

MD Zeiler, , , R Fergus. . Visualizing and understanding convolutional networks. . European Conf on Computer Vision, , 2014. . p.818--833. . DOI:10.1007/978-3-319-10590-1_53http://doi.org/10.1007/978-3-319-10590-1_53..

Q Zhang, , , R Cao, , , YN Wu, , , 等. . Growing interpretable part graphs on convnets via multi-shot learning. . Proc 30th AAAI Conf on Artificial Intelligence, , 2016. . p.2898--2906. . ..

Q Zhang, , , R Cao, , , YN Wu, , , 等. . Mining object parts from CNNs via active question-answering. . Proc IEEE Conf on Computer Vision and Pattern Recognition, , 2017a. . p.346--355. . DOI:10.1109/CVPR.2017.414http://doi.org/10.1109/CVPR.2017.414..

Q Zhang, , , R Cao, , , S Zhang, , , 等. . Interactively transferring CNN patterns for part localization. . http://arxiv.org/abs/1708.01783, , 2017b. ..

Q Zhang, , , W Wang, , , SC Zhu. . Examining CNN representations with respect to dataset bias. . Proc 32nd AAAI Conf on Artificial Intelligence, in press, , 2018a. ..

Q Zhang, , , R Cao, , , F Shi, , , 等. . Interpreting CNN knowledge via an explanatory graph. . Proc 32nd AAAI Conf on Artificial Intelligence, , 2018b. . p.2124--2132. . ..

Q Zhang, , , Y Yang, , , YN Wu, , , 等. . Interpreting CNNs via decision trees. . http://arxiv.org/abs/1802.00121, , 2018c. ..

Q Zhang, , , YN Wu, , , SC Zhu. . Interpretable convolutional neural networks. . Proc IEEE Conf on Computer Vision and Pattern Recognition, in press, , 2018d. ..

B Zhou, , , A Khosla, , , A Lapedriza, , , 等. . Object detectors emerge in deep scene CNNs. . http://arxiv.org/abs/1412.6856, , 2015. ..

LM Zintgraf, , , TSCT Adel, , , M Welling. . Visualizing deep neural network decisions: prediction difference analysis. . http://arxiv.org/abs/1702.04595, , 2017. ..

Views

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Quant 4.0: engineering quantitative investment with automated, explainable, and knowledge-driven artificial intelligence

Accurate estimation of 6-DoF tooth pose in 3D intraoral scans for dental applications using deep learning

Smart grid dispatch powered by deep learning: a survey

Improved deep learning aided key recovery framework: applications to large-state block ciphers

Deep unfolding based channel estimation for wideband terahertz near-field massive MIMO systems

Related Author

Jian GUO

Saizhuo WANG

Lionel M. NI

Heung-Yeung SHUM

Wanghui DING

Kaiwei SUN

Mengfei YU

Hangzheng LIN

Related Institution

The Hong Kong University of Science and Technology

IDEA Research, International Digital Economy Academy

Hangzhou Dental Hospital

Stomatology Hospital, School of Stomatology, Zhejiang University School of Medicine, Zhejiang Provincial Clinical Research Center for Oral Diseases, Key Laboratory of Oral Biomedical Research of Zhejiang Province, Cancer Center of Zhejiang University, Engineering Research Center of Oral Biomaterials and Devices of Zhejiang Province

Angel Align Inc

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰