针对无监督域自适应问题的深度逐层领域修正算法

李爽; 宋士吉; 吴澄

doi:10.1631/FITEE.1700774

Your Location：

Home >

Browse articles >

针对无监督域自适应问题的深度逐层领域修正算法

2018 Special Issue on Artificial Intelligence 2.0: Theories and Applications | Updated：2022-05-19

- 针对无监督域自适应问题的深度逐层领域修正算法
- Layer-wise domain correction for unsupervised domain adaptation
- 信息与电子工程前沿（英文） 2018年19卷第1期页码：91-103
- Affiliations：
  
  Automation Department, Tsinghua University, Beijing 100084, China
- Author bio：
  
  [ "Shuang LI, l-s12@mails.tsinghua.edu.cn" ]
  Shi-ji SONG, E-mail: shijis@mail.tsinghua.edu.cn
  [ "Cheng WU, wuc@tsinghua.edu.cn" ]
- Funds：
  
  Project supported by the National Key R&D Program of China (No. 2016YFB1200203) and the National Natural Science Foundation of China (Nos. 41427806 and 61273233)
- DOI：10.1631/FITEE.1700774
  中图分类号： TP183
- 收稿：2017-11-19，
  
  修回：2018-;1-19，
  
  纸质出版：2018-01
- Accepted：
Scan QR Code
李爽, 宋士吉, 吴澄. 针对无监督域自适应问题的深度逐层领域修正算法[J]. 信息与电子工程前沿（英文）, 2018,19(1):91-103.

Shuang LI, Shi-ji SONG, Cheng WU. Layer-wise domain correction for unsupervised domain adaptation[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(1): 91-103.
李爽, 宋士吉, 吴澄. 针对无监督域自适应问题的深度逐层领域修正算法[J]. 信息与电子工程前沿（英文）, 2018,19(1):91-103. DOI： 10.1631/FITEE.1700774.

Shuang LI, Shi-ji SONG, Cheng WU. Layer-wise domain correction for unsupervised domain adaptation[J]. Frontiers of Information Technology & Electronic Engineering, 2018, 19(1): 91-103. DOI： 10.1631/FITEE.1700774.

摘要

深度神经网络凭借强大的特征抽象能力，已成功应用在机器学习的多个领域。然而，传统深度网络假设训练样本和测试样本来自同一分布，这一假设在很多实际应用中并不成立。为借助深度网络解决领域偏移问题，本文提出逐层领域修正（layer-wise domain correction

LDC）深度域自适应算法。该算法通过在已有深度网络中增加领域修正层，将源域网络成功适配到目标领域。逐层增加的领域修正层能够将两个领域特征的最大均值偏差（maximum mean discrepancy

MMD）距离最小化，从而完美匹配源域和目标域样本的特征表示。与此同时，网络深度的增加极大提高了网络表达能力。LDC算法不需要目标领域有标记样本，在几个跨领域分类识别数据集都取得了当时最好结果，且其训练比已有深度域自适应算法快近10倍。

Abstract

Deep neural networks have been successfully applied to numerous machine learning tasks because of their impressive feature abstraction capabilities. However

conventional deep networks assume that the training and test data are sampled from the same distribution

and this assumption is often violated in real-world scenarios. To address the domain shift or data bias problems

we introduce layer-wise domain correction (LDC)

a new unsupervised domain adaptation algorithm which adapts an existing deep network through additive correction layers spaced throughout the network. Through the additive layers

the representations of source and target domains can be perfectly aligned. The corrections that are trained via maximum mean discrepancy

adapt to the target domain while increasing the representational capacity of the network. LDC requires no target labels

achieves state-of-the-art performance across several adaptation benchmarks

and requires significantly less training time than existing adaptation methods.

关键词

Keywords

references

H Ajakan , , , P Germain , , , H Larochelle , , , 等 . . Domainadversarial neural networks . . https://arxiv.org/abs/1412.4446 , , 2014 . . .

S Ben-David , , , J Blitzer , , , K Crammer , , , 等 . . A theory of learning from different domains . . Mach Learn , , 2010 . . 79 ( ( 1-2 ): ): 151 - - 175 . . DOI: 10.1007/s10994-009-5152-4 http://doi.org/10.1007/s10994-009-5152-4 . .

J Blitzer , , , R McDonald , , , F Pereira . . Domain adaptation with structural correspondence learning . . Proc Conf on Empirical Methods in Natural Language Processing , , 2006 . . p.120 - - 128 . . DOI: 10.3115/1610075.1610094 http://doi.org/10.3115/1610075.1610094 . .

KM Borgwardt , , , A Gretton , , , MJ Rasch , , , 等 . . Integrating structured biological data by kernel maximum mean discrepancy . . Bioinformatics , , 2006 . . 22 ( ( 14 ): ): e49 - - e57 . . DOI: 10.1093/bioinformatics/btl242 http://doi.org/10.1093/bioinformatics/btl242 . .

MM Chen , , , KQ Weinberger , , , JC Blitzer . . Co-training for domain adaptation . . Advances in Neural Information Processing Systems , , 2011 . . p.2456 - - 2464 . . . .

MM Chen , , , ZX Xu , , , K Weinberger , , , 等 . . Marginalized denoising autoencoders for domain adaptation . . https://arxiv.org/abs/1206.4683 , , 2012 . . .

J Donahue , , , YQ Jia , , , O Vinyals , , , 等 . . Decaf: a deep convolutional activation feature for generic visual recognition . . Proc 31 st Int Conf on Machine Learning , , 2014 . . p.647 - - 655 . . . .

LX Duan , , , IW Tsang , , , D Xu , , , 等 . . Domain transfer SVM for video concept detection . . IEEE Conf on Computer Vision and Pattern Recognition , , 2009 . . p.1375 - - 1381 . . DOI: 10.1109/CVPR.2009.5206747 http://doi.org/10.1109/CVPR.2009.5206747 . .

LX Duan , , , IW Tsang , , , D Xu . . Domain transfer multiple kernel learning . . IEEE Trans Patt Anal Mach Intell , , 2012 . . 34 ( ( 3 ): ): 465 - - 479 . . DOI: 10.1109/TPAMI.2011.114 http://doi.org/10.1109/TPAMI.2011.114 . .

Y Ganin , , , V Lempitsky . . Unsupervised domain adaptation by backpropagation . . Proc 32 nd Int Conf on Machine Learning , , 2015 . . p.1180 - - 1189 . . . .

JR Gardner , , , P Upchurch , , , MJ Kusner , , , 等 . . Deep manifold traversal: changing labels with convolutional features . . https://arxiv.org/abs/1511.06421 , , 2015 . . .

J Gehring , , , M Auli , , , D Grangier , , , 等 . . Convolutional sequence to sequence learning . . https://arxiv.org/abs/1705.03122 , , 2017 . . .

X Glorot , , , A Bordes , , , Y Bengio . . Domain adaptation for large-scale sentiment classification: a deep learning approach . . Proc 28 th Int Conf on Machine Learning , , 2011 . . p.513 - - 520 . . . .

BQ Gong , , , Y Shi , , , F Sha , , , 等 . . Geodesic flow kernel for unsupervised domain adaptation . . IEEE Conf on Computer Vision and Pattern Recognition , , 2012 . . p.2066 - - 2073 . . DOI: 10.1109/CVPR.2012.6247911 http://doi.org/10.1109/CVPR.2012.6247911 . .

BQ Gong , , , K Grauman , , , F Sha . . Connecting the dots with landmarks: discriminatively learning domaininvariant features for unsupervised domain adaptation . . Proc 30 th Int Conf on Machine Learning , , 2013 . . p.222 - - 230 . . . .

A Gretton , , , KM Borgwardt , , , MJ Rasch , , , 等 . . A kernel two-sample test . . J Mach Learn Res , , 2012 . . 13 ( ( 1 ): ): 723 - - 773 . . . .

KM He , , , XY Zhang , , , SQ Ren , , , 等 . . Delving deep into rectifiers: surpassing human-level performance on ImageNet classification . . IEEE Int Conf on Computer Vision , , 2015 . . p.1026 - - 1034 . . DOI: 10.1109/ICCV.2015.123 http://doi.org/10.1109/ICCV.2015.123 . .

KM He , , , XY Zhang , , , SQ Ren , , , 等 . . Deep residual learning for image recognition . . IEEE Conf on Computer Vision and Pattern Recognition , , 2016 . . p.770 - - 778 . . DOI: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90 . .

J Hoffman , , , E Tzeng , , , T Park , , , 等 . . CyCADA: cycleconsistent adversarial domain adaptation . . https://arxiv.org/abs/1711.03213 , , 2017 . . .

S Ioffe , , , C Szegedy . . Batch normalization: accelerating deep network training by reducing internal covariate shift . . Proc 32 nd Int Conf on Machine Learning , , 2015 . . p.448 - - 456 . . . .

DP Kingma , , , J Ba . . Adam: a method for stochastic optimization . . https://arxiv.org/abs/1412.6980 , , 2014 . . .

A Krizhevsky , , , I Sutskever , , , GE Hinton . . ImageNet classification with deep convolutional neural networks . . Commun ACM , , 2017 . . 60 ( ( 6 ): ): 84 - - 90 . . DOI: 10.1145/3065386 http://doi.org/10.1145/3065386 . .

Y LeCun , , , L Bottou , , , Y Bengio , , , 等 . . Gradient-based learning applied to document recognition . . Proc IEEE , , 1998 . . 86 ( ( 11 ): ): 2278 - - 2324 . . DOI: 10.1109/5.726791 http://doi.org/10.1109/5.726791 . .

YJ Li , , , K Swersky , , , R Zemel . . Generative moment matching networks . . Proc 32 nd Int Conf on Machine Learning , , 2015 . . p.1718 - - 1727 . . . .

MS Long , , , JM Wang , , , GG Ding , , , 等 . . Transfer feature learning with joint distribution adaptation . . Proc IEEE Int Conf on Computer Vision , , 2013 . . p.2200 - - 2207 . . DOI: 10.1109/ICCV.2013.274 http://doi.org/10.1109/ICCV.2013.274 . .

MS Long , , , JM Wang , , , GG Ding , , , 等 . . Transfer joint matching for unsupervised domain adaptation . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2014 . . p.1410 - - 1417 . . DOI: 10.1109/CVPR.2014.183 http://doi.org/10.1109/CVPR.2014.183 . .

MS Long , , , Y Cao , , , JM Wang , , , 等 . . Learning transferable features with deep adaptation networks . . Proc 32 nd Int Conf on Machine Learning , , 2015 . . p.97 - - 105 . . . .

MS Long , , , JM Wang , , , Y Cao , , , 等 . . Deep learning of transferable representation for scalable domain adaptation . . IEEE Trans Knowl Data Eng , , 2016a . . 28 ( ( 8 ): ): 2027 - - 2040 . . DOI: 10.1109/TKDE.2016.2554549 http://doi.org/10.1109/TKDE.2016.2554549 . .

MS Long , , , H Zhu , , , JM Wang , , , 等 . . Unsupervised domain adaptation with residual transfer networks . . Advances in Neural Information Processing Systems , , 2016b . . p.136 - - 144 . . . .

T Mikolov , , , I Sutskever , , , K Chen , , , 等 . . Distributed representations of words and phrases and their compositionality . . Proc 26 th Int Conf on Neural Information Processing Systems , , 2013 . . p.3111 - - 3119 . . . .

Y Netzer , , , T Wang , , , A Coates , , , 等 . . Reading digits in natural images with unsupervised feature learning . . NIPS Workshop on Deep Learning and Unsupervised Feature Learning , , 2011 . . p.1 - - 9 . . . .

M Oquab , , , L Bottou , , , I Laptev , , , 等 . . Learning and transferring mid-level image representations using convolutional neural networks . . Proc IEEE Conf on Computer Vision and Pattern Recognition , , 2014 . . p.1717 - - 1724 . . DOI: 10.1109/CVPR.2014.222 http://doi.org/10.1109/CVPR.2014.222 . .

SJL Pan , , , Q Yang . . A survey on transfer learning . . IEEE Trans Knowl Data Eng , , 2010 . . 22 ( ( 10 ): ): 1345 - - 1359 . . DOI: 10.1109/TKDE.2009.191 http://doi.org/10.1109/TKDE.2009.191 . .

SJL Pan , , , IW Tsang , , , JT Kwok , , , 等 . . Domain adaptation via transfer component analysis . . IEEE Trans Neur Netw , , 2011 . . 22 ( ( 2 ): ): 199 - - 210 . . DOI: 10.1109/TNN.2010.2091281 http://doi.org/10.1109/TNN.2010.2091281 . .

O Russakovsky , , , J Deng , , , H Su , , , 等 . . ImageNet large scale visual recognition challenge . . Int J Comput Vis , , 2015 . . 115 ( ( 3 ): ): 211 - - 252 . . DOI: 10.1007/s11263-015-0816-y http://doi.org/10.1007/s11263-015-0816-y . .

K Saenko , , , B Kulis , , , M Fritz , , , 等 . . Adapting visual category models to new domains . . LNCS , , 2010 . . 6314 213 - - 226 . . DOI: 10.1007/978-3-642-15561-1_16 http://doi.org/10.1007/978-3-642-15561-1_16 . .

K Simonyan , , , A Zisserman . . Very deep convolutional networks for large-scale image recognition . . https://arxiv.org/abs/1409.1556 , , 2014 . . .

N Srivastava , , , G Hinton , , , A Krizhevsky , , , 等 . . Dropout: a simple way to prevent neural networks from overfitting . . J Mach Learn Res , , 2014 . . 15 ( ( 1 ): ): 1929 - - 1958 . . . .

I Sutskever , , , J Martens , , , G Dahl , , , 等 . . On the importance of initialization and momentum in deep learning . . Proc 30 th Int Conf on Machine Learning , , 2013 . . p.1139 - - 1147 . . . .

I Sutskever , , , O Vinyals , , , Q Le . . Sequence to sequence learning with neural networks . . Advances in Neural Information Processing Systems , , 2014 . . p.3104 - - 3112 . . . .

E Tzeng , , , J Hoffman , , , N Zhang , , , 等 . . Deep domain confusion: maximizing for domain invariance . . https://arxiv.org/abs/1412.3474 , , 2014 . . .

L van der Maaten , , , G Hinton . . Visualizing data using t-SNE . . J Mach Learn Res , , 2008 . . 9 ( ( 11 ): ): 2579 - - 2605 . . . .

J Yosinski , , , J Clune , , , Y Bengio , , , 等 . . How transferable are features in deep neural networks . . Proc 27 th Int Conf on Neural Information Processing Systems , , 2014 . . p.3320 - - 3328 . . . .

浏览量

Downloads

CSCD

文章被引用时，请邮件提醒。

Submit

工具集

关联资源

MltAuxTSPP: a unified benchmark for deep learning-based traffic state prediction with multi-source auxiliary data

Q-space-coordinate-guided neural networks for high-fidelity diffusion tensor estimation from minimal diffusion-weighted images

CRGT-SA: an interlaced and spatiotemporal deep learning model for network intrusion detection

Federated model with contrastive learning and adaptive control variates for human activity recognition

A comprehensive survey of physical adversarial vulnerabilities in autonomous driving systems