Federated unsupervised representation learning

Fengda ZHANG; Kun KUANG; Long CHEN; Zhaoyang YOU; Tao SHEN; Jun XIAO; Yin ZHANG; Chao WU; Fei WU; Yueting ZHUANG; Xiaolin LI

doi:10.1631/FITEE.2200268

Your Location：

Home >

Browse articles >

Federated unsupervised representation learning

Regular Papers | Updated：2023-08-29

- Federated unsupervised representation learning
- 联邦无监督表示学习
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 8, Pages: 1181-1193(2023)
- Affiliations：
  
  1.College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  2.School of Public Affairs, Zhejiang University, Hangzhou 310027, China
  3.Tongdun Technology, Hangzhou 310000, China
  4.Institute of Basic Medicine and Cancer, Chinese Academy of Sciences, Hangzhou 310018, China
  5.ElasticMind.AI Technology Inc., Hangzhou 310018
- Author bio：
  
  †E-mail: fdzhang@zju.edu.cn
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2200268
  CLC： TP183
- Received：21 June 2022，
  
  Accepted：27 October 2022，
  
  Published：0 August 2023
- Accepted：
Scan QR Code
Fengda ZHANG, Kun KUANG, Long CHEN, et al. Federated unsupervised representation learning[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(8): 1181-1193.
DOI：

Fengda ZHANG, Kun KUANG, Long CHEN, et al. Federated unsupervised representation learning[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(8): 1181-1193. DOI： 10.1631/FITEE.2200268.

摘要

为利用分布式边缘设备上大量未标记数据，我们在联邦学习中提出一个称为联邦无监督表示学习（FURL）的新问题，以在没有监督的情况下学习通用表示模型，同时保护数据隐私。FURL提出了两个新挑战：（1）客户端之间的数据分布转移（非独立同分布）会使本地模型专注于不同的类别，从而导致表示空间的不一致；（2）如果FURL中客户端之间没有统一的信息，客户端之间的表示就会错位。为了应对这些挑战，我们提出带字典和对齐的联合对比平均（FedCA）算法。FedCA由两个关键模块组成：字典模块，用于聚合来自每个客户端的样本表示并与所有客户端共享，以实现表示空间的一致性；对齐模块，用于将每个客户端的表示与基于公共数据训练的基础模型对齐。我们采用对比方法进行局部模型训练，通过在3个数据集上独立同分布和非独立同分布设定下的大量实验，我们证明FedCA以显著的优势优于所有基线方法。

Abstract

To leverage the enormous amount of unlabeled data on distributed edge devices

we formulate a new problem in federated learning called federated unsupervised representation learning (FURL) to learn a common representation model without supervision while preserving data privacy. FURL poses two new challenges: (1) data distribution shift (non-independent and identically distributed

non-IID) among clients would make local models focus on different categories

leading to the inconsistency of representation spaces; (2) without unified information among the clients in FURL

the representations across clients would be misaligned. To address these challenges

we propose the federated contrastive averaging with dictionary and alignment (FedCA) algorithm. FedCA is composed of two key modules: a dictionary module to aggregate the representations of samples from each client which can be shared with all clients for consistency of representation space and an alignment module to align the representation of each client on a base model trained on public data. We adopt the contrastive approach for local model training. Through extensive experiments with three evaluation protocols in IID and non-IID settings

we demonstrate that FedCA outperforms all baselines with significant margins.

关键词

Keywords

references

Baevski A , Zhou H , Mohamed A , et al. , 2020 . wav2vec 2.0: a framework for self-supervised learning of speech representations . Proc 34 th Conf on Neural Information Processing Systems .

Bonawitz K , Ivanov V , Kreuter B , et al. , 2017 . Practical secure aggregation for privacy-preserving machine learning . Proc ACM SIGSAC Conf on Computer and Communications Security , p. 1175 - 1191 . doi: 10.1145/3133956.3133982 http://doi.org/10.1145/3133956.3133982

Chen T , Kornblith S , Norouzi M , et al. , 2020 . A simple framework for contrastive learning of visual representations . Proc 37 th Int Conf on Machine Learning , Article 149 .

Chen XL , Fan HQ , Girshick R , et al. , 2020 . Improved baselines with momentum contrastive learning . https://arxiv.org/abs/2003.04297 https://arxiv.org/abs/2003.04297

Coates A , Ng AY , Lee H , 2011 . An analysis of single-layer networks in unsupervised feature learning . Proc 14 th Int Conf on Artificial Intelligence and Statistics , p. 215 - 223 .

Deng J , Dong W , Socher R , et al. , 2009 . ImageNet: a large-scale hierarchical image database . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 248 - 255 . doi: 10.1109/CVPR.2009.5206848 http://doi.org/10.1109/CVPR.2009.5206848

Dosovitskiy A , Springenberg JT , Riedmiller M , et al. , 2014 . Discriminative unsupervised feature learning with convolutional neural networks . Proc 27 th Int Conf on Neural Information Processing Systems , p. 766 - 774 .

Duan XY , Tang SL , Zhang SY , et al. , 2018 . Temporality-enhanced knowledge memory network for factoid question answering . Front Inform Technol Electron Eng , 19 ( 1 ): 104 - 115 . doi: 10.1631/FITEE.1700788 http://doi.org/10.1631/FITEE.1700788

Gidaris S , Singh P , Komodakis N , 2018 . Unsupervised representation learning by predicting image rotations . Proc 6 th Int Conf on Learning Representations .

Hadsell R , Chopra S , LeCun Y , 2006 . Dimensionality reduction by learning an invariant mapping . Proc IEEE Computer Society Conf on Computer Vision and Pattern Recognition , p. 1735 - 1742 . doi: 10.1109/CVPR.2006.100 http://doi.org/10.1109/CVPR.2006.100

Hassani K , Ahmadi AHK , 2020 . Contrastive multi-view representation learning on graphs . Proc 37 th Int Conf on Machine Learning , p. 4116 - 4126 .

He CY , Yang ZY , Mushtaq E , et al. , 2021 . SSFL: tackling label deficiency in federated learning via personalized self-supervision . https://arxiv.org/abs/2110.02470 https://arxiv.org/abs/2110.02470

He KM , Zhang XY , Ren SQ , et al. , 2016 . Deep residual learning for image recognition . Proc IEEE Conf on Computer Vision and Pattern Recognition , p. 770 - 778 . doi: 10.1109/CVPR.2016.90 http://doi.org/10.1109/CVPR.2016.90

He KM , Fan HQ , Wu YX , et al. , 2020 . Momentum contrast for unsupervised visual representation learning . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 9729 - 9738 . doi: 10.1109/CVPR42600.2020.00975 http://doi.org/10.1109/CVPR42600.2020.00975

Hinton GE , Salakhutdinov RR , 2006 . Reducing the dimensionality of data with neural networks . Science , 313 ( 5786 ): 504 - 507 . doi: 10.1126/science.1127647 http://doi.org/10.1126/science.1127647

Jeong E , Oh S , Kim H , et al. , 2018 . Communication-efficient on-device machine learning: federated distillation and augmentation under non-IID private data . https://arxiv.org/abs/1811.11479v1 https://arxiv.org/abs/1811.11479v1

Ji SX , Saravirta T , Pan SR , et al. , 2021 . Emerging trends in federated learning: from model fusion to federated X learning . https://arxiv.org/abs/2102.12920 https://arxiv.org/abs/2102.12920

Jin YL , Wei XG , Liu Y , et al. , 2020 . Towards utilizing unlabeled data in federated learning: a survey and prospective . https://arxiv.org/abs/2002.11545 https://arxiv.org/abs/2002.11545

Kairouz P , McMahan HB , Avent B , et al. , 2021 . Advances and open problems in federated learning . https://arxiv.org/abs/1912.04977 https://arxiv.org/abs/1912.04977

Kempe D , McSherry F , 2008 . A decentralized algorithm for spectral analysis . J Comput Syst Sci , 74 ( 1 ): 70 - 83 . doi: 10.1016/j.jcss.2007.04.014 http://doi.org/10.1016/j.jcss.2007.04.014

Kingma DP , Welling M , 2014 . Auto-encoding variational Bayes . Proc 2 nd Int Conf on Learning Representations .

Konečný J , McMahan HB , Yu FX , et al. , 2017 . Federated learning: strategies for improving communication efficiency . https://arxiv.org/abs/1610.05492 https://arxiv.org/abs/1610.05492

Krizhevsky A , 2009 . Learning Multiple Layers of Features from Tiny Images . Technical Report TR-2009 , University of Toronto , Toronto, Canada .

Krizhevsky A , Sutskever I , Hinton GE , 2012 . ImageNet classification with deep convolutional neural networks . Proc 25 th Int Conf on Neural Information Processing Systems , p. 1097 - 1105 .

Kuang K , Li L , Geng Z , et al. , 2020 . Causal inference . Engineering , 6 ( 3 ): 253 - 263 . doi: 10.1016/j.eng.2019.08.016 http://doi.org/10.1016/j.eng.2019.08.016

Lei N , An DS , Guo Y , et al. , 2020 . A geometric understanding of deep learning . Engineering , 6 ( 3 ): 361 - 374 . doi: 10.1016/j.eng.2019.09.010 http://doi.org/10.1016/j.eng.2019.09.010

Li QB , He BS , Song D , 2021 . Model-contrastive federated learning . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 10713 - 10722 . doi: 10.1109/CVPR46437.2021.01057 http://doi.org/10.1109/CVPR46437.2021.01057

Li T , Sahu AK , Zaheer M , et al. , 2020 . Federated optimization in heterogeneous networks . Proc 3 rd MLSys Conf .

Liang JL , Zhang MH , Zeng XY , et al. , 2014 . Distributed dictionary learning for sparse representation in sensor networks . IEEE Trans Image Process , 23 ( 6 ): 2528 - 2541 . doi: 10.1109/TIP.2014.2316373 http://doi.org/10.1109/TIP.2014.2316373

Logeswaran L , Lee H , 2018 . An efficient framework for learning sentence representations . Proc 6 th Int Conf on Learning Representations .

Lyu YG , 2020 . Artificial intelligence: enabling technology to empower society . Engineering , 6 ( 3 ): 205 - 206 . doi: 10.1016/j.eng.2020.01.005 http://doi.org/10.1016/j.eng.2020.01.005

McMahan B , Moore E , Ramage D , et al. , 2017 . Communication-efficient learning of deep networks from decentralized data . Proc 20 th Int Conf on Artificial Intelligence and Statistics , p. 1273 - 1282 .

Mikolov T , Sutskever I , Chen K , et al. , 2013 . Distributed representations of words and phrases and their compositionality . Proc 26 th Int Conf on Neural Information Processing Systems , p. 3111 - 3119 .

Pan YH , 2020 . Multiple knowledge representation of artificial intelligence . Engineering , 6 ( 3 ): 216 - 217 . doi: 10.1016/j.eng.2019.12.011 http://doi.org/10.1016/j.eng.2019.12.011

Paszke A , Gross S , Massa F , et al. , 2019 . PyTorch: an imperative style, high-performance deep learning library . Proc 33 rd Conf on Neural Information Processing Systems , p. 8026 - 8037 .

Pathak D , Agrawal P , Efros AA , et al. , 2017 . Curiosity-driven exploration by self-supervised prediction . Proc IEEE Conf on Computer Vision and Pattern Recognition Workshops , p. 16 - 17 . doi: 10.1109/CVPRW.2017.70 http://doi.org/10.1109/CVPRW.2017.70

Qiu JZ , Chen QB , Dong YX , et al. , 2020 . GCC: graph contrastive coding for graph neural network pre-training . Proc 26 th ACM SIGKDD Int Conf on Knowledge Discovery & Data Mining , p. 1150 - 1160 . doi: 10.1145/3394486.3403168 http://doi.org/10.1145/3394486.3403168

Radford A , Metz L , Chintala S , 2016 . Unsupervised representation learning with deep convolutional generative adversarial networks . Proc 4 th Int Conf on Learning Representations .

Raja H , Bajwa WU , 2016 . Cloud K-SVD: a collaborative dictionary learning algorithm for big, distributed data . IEEE Trans Signal Process , 64 ( 1 ): 173 - 188 . doi: 10.1109/TSP.2015.2472372 http://doi.org/10.1109/TSP.2015.2472372

Sattler F , Wiedemann S , Müller KR , et al. , 2020 . Robust and communication-efficient federated learning from non-i.i.d. data . IEEE Trans Neur Netw Learn Syst , 31 ( 9 ): 3400 - 3413 . doi: 10.1109/TNNLS.2019.2944481 http://doi.org/10.1109/TNNLS.2019.2944481

Sattler F , Korjakow T , Rischke R , et al. , 2021 . FEDAUX: leveraging unlabeled auxiliary data in federated learning . IEEE Trans Neur Netw Learn Syst, early access . doi: 10.1109/TNNLS.2021.3129371 http://doi.org/10.1109/TNNLS.2021.3129371

Sermanet P , Lynch C , Chebotar Y , et al. , 2018 . Time-contrastive networks: self-supervised learning from video . Proc IEEE Int Conf on Robotics and Automation , p. 1134 - 1141 . doi: 10.1109/ICRA.2018.8462891 http://doi.org/10.1109/ICRA.2018.8462891

Shakeri Z , Raja H , Bajwa WU , 2014 . Dictionary learning based nonlinear classifier training from distributed data . Proc IEEE Global Conf on Signal and Information Processing , p. 759 - 763 . doi: 10.1109/GlobalSIP.2014.7032221 http://doi.org/10.1109/GlobalSIP.2014.7032221

Shi HZ , Zhang YC , Shen ZJ , et al. , 2022 . Federated self-supervised contrastive learning via ensemble similarity distillation . https://arxiv.org/abs/2109.14611v1 https://arxiv.org/abs/2109.14611v1

Sohn K , 2016 . Improved deep metric learning with multi-class N -pair loss objective . Proc 30 th Int Conf on Neural Information Processing Systems , p. 1857 - 1865 .

Tian YL , Krishnan D , Isola P , 2020 . Contrastive multiview coding . Proc 16 th European Conf on Computer Vision , p. 776 - 794 . doi: 10.1007/978-3-030-58621-8_45 http://doi.org/10.1007/978-3-030-58621-8_45

van Berlo B , Saeed A , Ozcelebi T , 2020 . Towards federated unsupervised representation learning . Proc 3 rd ACM Int Workshop on Edge Systems, Analytics and Networking , p. 31 - 36 . doi: 10.1145/3378679.3394530 http://doi.org/10.1145/3378679.3394530

van den Oord A , Li YZ , Vinyals O , 2019 . Representation learning with contrastive predictive coding . https://arxiv.org/abs/1807.03748 https://arxiv.org/abs/1807.03748

Vinyals O , Blundell C , Lillicrap T , et al. , 2016 . Matching networks for one shot learning . Proc 30 th Int Conf on Neural Information Processing Systems , p. 3637 - 3645 .

Wang HY , Yurochkin M , Sun YK , et al. , 2020 . Federated learning with matched averaging . Proc 8 th Int Conf on Learning Representations .

Wang TZ , Isola P , 2020 . Understanding contrastive representation learning through alignment and uniformity on the hypersphere . Proc 37 th Int Conf on Machine Learning , p. 9929 - 9939 .

Wu SX , Wai HT , Li L , et al. , 2018 . A review of distributed algorithms for principal component analysis . Proc IEEE , 106 ( 8 ): 1321 - 1340 . doi: 10.1109/JPROC.2018.2846568 http://doi.org/10.1109/JPROC.2018.2846568

Wu YW , Zeng DW , Wang ZP , et al. , 2021 . Federated contrastive learning for volumetric medical image segmentation . Proc 24 th Int Conf on Medical Image Computing and Computer-Assisted Intervention , p. 367 - 377 . doi: 10.1007/978-3-030-87199-4_35 http://doi.org/10.1007/978-3-030-87199-4_35

Wu ZR , Xiong YJ , Yu SX , et al. , 2018 . Unsupervised feature learning via non-parametric instance discrimination . Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 3733 - 3742 . doi: 10.1109/CVPR.2018.00393 http://doi.org/10.1109/CVPR.2018.00393

Yang Q , Liu Y , Chen TJ , et al. , 2019 . Federated machine learning: concept and applications . ACM Trans Intell Syst Technol , 10 ( 2 ): 12 . doi: 10.1145/3298981 http://doi.org/10.1145/3298981

Yang ZL , Dai ZH , Yang YM , et al. , 2019 . XLNet: generalized autoregressive pretraining for language understanding . Proc 33 rd Int Conf on Neural Information Processing Systems , Article 517 .

Zhao Y , Li M , Lai LZ , et al. , 2022 . Federated learning with non-IID data . https://arxiv.org/abs/1806.00582 https://arxiv.org/abs/1806.00582

Zhou LK , Tang SL , Xiao J , et al. , 2017 . Disambiguating named entities with deep supervised learning via crowd labels . Front Inform Technol Electron Eng , 18 ( 1 ): 97 - 106 . doi: 10.1631/FITEE.1601835 http://doi.org/10.1631/FITEE.1601835

Zhu YX , Gao T , Fan LF , et al. , 2020 . Dark, beyond deep: a paradigm shift to cognitive AI with humanlike common sense . Engineering , 6 ( 3 ): 310 - 345 . doi: 10.1016/j.eng.2020.01.011 http://doi.org/10.1016/j.eng.2020.01.011

Zhuang WM , Gan X , Wen YG , et al. , 2021a . Collaborative unsupervised visual representation learning from decentralized data . Proc IEEE/CVF Int Conf on Computer Vision , p. 4892 - 4901 . doi: 10.1109/ICCV48922.2021.00487 http://doi.org/10.1109/ICCV48922.2021.00487

Zhuang WM , Wen YG , Zhang S , 2021b . Joint optimization in edge-cloud continuum for federated unsupervised person re-identification . Proc 29 th ACM Int Conf on Multimedia , p. 433 - 441 . doi: 10.1145/3474085.3475182 http://doi.org/10.1145/3474085.3475182

Zhuang WM , Wen YG , Zhang S , 2022 . Divergence-aware federated self-supervised learning . Proc 10 th Int Conf on Learning Representations .

Zhuang YT , Wu F , Chen C , et al. , 2017 . Challenges and opportunities: from big data to knowledge in AI 2.0 . Front Inform Technol Electron Eng , 18 ( 1 ): 3 - 14 . doi: 10.1631/FITEE.1601883 http://doi.org/10.1631/FITEE.1601883

Zhuang YT , Cai M , Li XL , et al. , 2020 . The next breakthroughs of artificial intelligence: the interdisciplinary nature of AI . Engineering , 6 ( 3 ): 245 - 247 . doi: 10.1016/j.eng.2020.01.009 http://doi.org/10.1016/j.eng.2020.01.009

Views

263

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Prototypical clustered federated learning for heart rate prediction

FedMcon: an adaptive aggregation method for federated learning via meta controller^&

Federated model with contrastive learning and adaptive control variates for human activity recognition

Federated deep reinforcement learning based computation offloading in a low Earth orbit satellite edge computing system

Toward an accurate mobility trajectory recovery using contrastive learning

Related Author

Qingyuan GONG

Yipeng ZHOU

Xiang SU

Ziyue LI

Jiong CHEN

Yang CHEN

Hui RUAN

Yongjie YIN

Related Institution

Research Institute of Intelligent Complex Systems, Fudan University

School of Computing, Macquarie University

Department of Agricultural Sciences, University of Helsinki

Information System Department, University of Cologne

NIO

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰