Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives

Tao SHEN; Jie ZHANG; Xinkang JIA; Fengda ZHANG; Zheqi LV; Kun KUANG; Chao WU; Fei WU

doi:10.1631/FITEE.2300098

Your Location：

Home >

Browse articles >

Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives

Regular Papers | Updated：2023-10-25

- Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives
- 联邦相互学习：一种针对异构数据、模型和目标的协同机器学习方法
- Frontiers of Information Technology & Electronic Engineering Vol. 24, Issue 10, Pages: 1390-1402(2023)
- Affiliations：
  
  1.College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
  2.School of Software Technology, Zhejiang University, Hangzhou 310027, China
  3.School of Public Affairs, Zhejiang University, Hangzhou 310027, China
- Author bio：
  
  ‡Corresponding author
- Funds：
- DOI：10.1631/FITEE.2300098
  CLC： TP39
- Received：20 February 2023，
  
  Accepted：07 April 2023，
  
  Published Online：05 August 2023，
  
  Published：0 October 2023
- Accepted：
Scan QR Code
Tao SHEN, Jie ZHANG, Xinkang JIA, et al. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(10): 1390-1402.
DOI：

Tao SHEN, Jie ZHANG, Xinkang JIA, et al. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives[J]. Frontiers of Information Technology & Electronic Engineering, 2023, 24(10): 1390-1402. DOI： 10.1631/FITEE.2300098.

摘要

联邦学习（FL）是深度学习中的一种新技术，可以让客户端在保留各自隐私数据的情况下协同训练模型。然而，由于每个客户端的数据分布、算力和场景都不同，联邦学习面临客户端异构环境的挑战。现有方法（如FedAvg）无法有效满足每个客户的定制化需求。为解决联邦学习中的异构挑战，本文首先详述了数据、模型和目标（DMO）这3个主要异构来源，然后提出一种新的联邦相互学习（FML）框架。该框架使得每个客户端都能训练一个考虑到数据异构（DH）的个性化模型。在模型异构（MH）问题上，引入一种“模因模型”作为个性化模型与全局模型之间的中介，并且采用深度相互学习（DML）的知识蒸馏技术在两个异构模型之间传递知识。针对目标异构（OH）问题，通过共享部分模型参数，设计针对特定任务的个性化模型，同时，利用模因模型进行相互学习。本研究通过实验评估了FML在应对DMO异构性方面的表现，并与其他常见FL方法在相似场景下进行对比。实验结果表明，FML在处理FL环境中的DMO问题的表现卓越，优于其他方法。

Abstract

Federated learning (FL) is a novel technique in deep learning that enables clients to collaboratively train a shared model while retaining their decentralized data. However

researchers working on FL face several unique challenges

especially in the context of heterogeneity. Heterogeneity in data distributions

computational capabilities

and scenarios among clients necessitates the development of customized models and objectives in FL. Unfortunately

existing works such as FedAvg may not effectively accommodate the specific needs of each client. To address the challenges arising from heterogeneity in FL

we provide an overview of the heterogeneities in data

model

and objective (DMO). Furthermore

we propose a novel framework called federated mutual learning (FML)

which enables each client to train a personalized model that accounts for the data heterogeneity (DH). A "meme model" serves as an intermediary between the personalized and global models to address model heterogeneity (MH). We introduce a knowledge distillation technique called deep mutual learning (DML) to transfer knowledge between these two models on local data. To overcome objective heterogeneity (OH)

we design a shared global model that includes only certain parts

and the personalized model is task-specific and enhanced through mutual learning with the meme model. We evaluate the performance of FML in addressing DMO heterogeneities through experiments and compare it with other commonly used FL methods in similar scenarios. The results demonstrate that FML outperforms other methods and effectively addresses the DMO challenges encountered in the FL setting.

关键词

Keywords

references

Alam S , Liu LY , Yan M , et al. , 2023 . FedRolex: model-heterogeneous federated learning with rolling sub-model extraction . https://arxiv.org/abs/2212.01548 https://arxiv.org/abs/2212.01548

Chen HT , Wang YH , Xu C , et al. , 2019 . Data-free learning of student networks . IEEE/CVF Int Conf on Computer Vision , p. 3513 - 3521 . 10.1109/ICCV.2019.00361 https://doi.org/10.1109/ICCV.2019.00361

Chen HY , Chao WL , 2022 . On bridging generic and personalized federated learning for image classification . https://arxiv.org/abs/2107.00778 https://arxiv.org/abs/2107.00778

Corchado JM , Li WG , Bajo J , et al. , 2016 . Special issue on distributed computing and artificial intelligence . Front Inform Technol Electron Eng , 17 ( 4 ): 281 - 282 . 10.1631/FITEE.DCAI2015 https://doi.org/10.1631/FITEE.DCAI2015

Gao DS , Ju C , Wei XG , et al. , 2020 . HHHFL: hierarchical heterogeneous horizontal federated learning for electroencephalography . https://arxiv.org/abs/1909.05784 https://arxiv.org/abs/1909.05784

Gao JQ , Li JQ , Shan HM , et al. , 2023 . Forget less, count better: a domain-incremental self-distillation learning benchmark for lifelong crowd counting . Front Inform Technol Electron Eng , 24 ( 2 ): 187 - 202 . 10.1631/FITEE.2200380 https://doi.org/10.1631/FITEE.2200380

He CY , Annavaram M , Avestimehr S , et al. , 2021 . FedNAS: federated deep learning via neural architecture search . https://arxiv.org/abs/2004.08546v1 https://arxiv.org/abs/2004.08546v1

Hinton G , Vinyals O , Dean J , 2015 . Distilling the knowledge in a neural network . https://arxiv.org/abs/1503.02531 https://arxiv.org/abs/1503.02531

Jiang YH , Konečný J , Rush K , et al. , 2023 . Improving federated learning personalization via model agnostic meta learning . https://arxiv.org/abs/1909.12488 https://arxiv.org/abs/1909.12488

Kairouz P , McMahan HB , Avent B , et al. , 2021 . Advances and open problems in federated learning . Found Trends Mach Learn , 14 ( 1-2 ): 1 - 210 . 10.1561/2200000083 https://doi.org/10.1561/2200000083

Khodak M , Balcan MF , Talwalkar A , 2019 . Adaptive gradient-based meta-learning methods . https://arxiv.org/abs/1906.02717 https://arxiv.org/abs/1906.02717

Krizhevsky A , 2009 . Learning Multiple Layers of Features from Tiny Images . Master Thesis , Department of Computer Science , University of Toronto, Canada .

LeCun Y , Boser B , Denker J , et al. , 1989 . Handwritten digit recognition with a back-propagation network . Proc 2 nd Int Conf on Neural Information Processing Systems , p. 396 - 404 .

LeCun Y , Bottou L , Bengio Y , et al. , 1998 . Gradient-based learning applied to document recognition . Proc IEEE , 86 ( 11 ): 2278 - 2324 . 10.1109/5.726791 https://doi.org/10.1109/5.726791

Li DL , Wang JP , 2019 . FedMD: heterogenous federated learning via model distillation . https://arxiv.org/abs/1910.03581 https://arxiv.org/abs/1910.03581

Li JH , 2018 . Cyber security meets artificial intelligence: a survey . Front Inform Technol Electron Eng , 19 ( 12 ): 1462 - 1474 . 10.1631/FITEE.1800573 https://doi.org/10.1631/FITEE.1800573

Li T , Sahu AK , Zaheer M , et al. , 2020 . Federated optimization in heterogeneous networks . https://arxiv.org/abs/1812.06127v5 https://arxiv.org/abs/1812.06127v5

Li WH , Bilen H , 2020 . Knowledge distillation for multi-task learning . Proc European Conf on Computer Vision , p. 163 - 176 . 10.1007/978-3-030-65414-6_13 https://doi.org/10.1007/978-3-030-65414-6_13

Li X , Huang KX , Yang WH , et al. , 2019 . On the convergence of FedAvg on non-IID data . https://arxiv.org/abs/1907.02189 https://arxiv.org/abs/1907.02189

Li X , Yang WH , Wang SS , et al. , 2021 . Communication efficient decentralized training with multiple local updates . https://arxiv.org/abs/1910.09126v1 https://arxiv.org/abs/1910.09126v1

Lian XR , Zhang C , Zhang H , et al. , 2017 . Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent . Proc 31 st Int Conf on Neural Information Processing Systems , p. 5336 - 5346 .

Liang PP , Liu T , Liu ZY , et al. , 2020 . Think locally, act globally: federated learning with local and global representations . https://arxiv.org/abs/2001.01523 https://arxiv.org/abs/2001.01523

Lim WYB , Luong NC , Hoang DT , et al. , 2020 Federated learning in mobile edge networks: a comprehensive survey . IEEE Commun Surv Tutor , 22 ( 3 ): 2031 - 2063 . 10.1109/COMST.2020.2986024 https://doi.org/10.1109/COMST.2020.2986024

Liu FL , Wu X , Ge S , et al. , 2020 . Federated learning for vision-and-language grounding problems . Proc AAAI Conf Artif Intell , 34 ( 7 ): 11572 - 11579 . 10.1609/aaai.v34i07.6824 https://doi.org/10.1609/aaai.v34i07.6824

Liu PX , Jiang JM , Zhu GX , et al. , 2022 . Training time minimization for federated edge learning with optimized gradient quantization and bandwidth allocation . Front Inform Technol Electron Eng , 23 ( 8 ): 1247 - 1263 . 10.1631/FITEE.2100538 https://doi.org/10.1631/FITEE.2100538

McMahan B , Moore E , Ramage D , et al. , 2017 . Communication-efficient learning of deep networks from decentralized data . Proc 20 th Int Conf on Artificial Intelligence and Statistics , p. 1273 - 1282 .

Padhya M , Jinwala DC , 2019 . MULKASE: a novel approach for key-aggregate searchable encryption for multi-owner data . Front Inform Technol Electron Eng , 20 ( 12 ): 1717 - 1748 . 10.1631/FITEE.1800192 https://doi.org/10.1631/FITEE.1800192

Pan YH , 2017 . Special issue on artificial intelligence 2.0 . Front Inform Technol Electron Eng , 18 ( 1 ): 1 - 2 . 10.1631/FITEE.1710000 https://doi.org/10.1631/FITEE.1710000

Pan YH , 2018 . 2018 special issue on artificial intelligence 2.0: theories and applications . Front Inform Technol Electron Eng , 19 ( 1 ): 1 - 2 . 10.1631/FITEE.1810000 https://doi.org/10.1631/FITEE.1810000

Smith V , Chiang CK , Sanjabi M , et al. , 2017 . Federated multi-task learning . Proc 31 st Int Conf on Neural Information Processing Systems , p. 4427 - 4437 .

Wang J , Li R , Wang J , et al. , 2020 . Artificial intelligence and wireless communications . Front Inform Technol Electron Eng , 21 ( 10 ): 1413 - 1425 . 10.1631/FITEE.1900527 https://doi.org/10.1631/FITEE.1900527

Wang TZ , Zhu JY , Torralba A , et al. , 2020 . Dataset distillation . https://arxiv.org/abs/1811.10959 https://arxiv.org/abs/1811.10959

Wu BC , Dai XL , Zhang PZ , et al. , 2019 . FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search . IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 10726 - 10734 . 10.1109/CVPR.2019.01099 https://doi.org/10.1109/CVPR.2019.01099

Wu JX , Li JH , Ji XS , 2018 . Security for cyberspace: challenges and opportunities . Front Inform Technol Electron Eng , 19 ( 12 ): 1459 - 1461 . 10.1631/FITEE.1840000 https://doi.org/10.1631/FITEE.1840000

Yang Q , Liu Y , Cheng Y , et al. , 2019 . Federated Learning . Springer , Cham, Switzerland , p. 1 - 207 .

Yu T , Bagdasaryan E , Shmatikov V , 2022 . Salvaging federated learning by local adaptation . https://arxiv.org/abs/2002.04758 https://arxiv.org/abs/2002.04758

Zhang X , Li YC , Li WP , et al. , 2022 . Personalized federated learning via variational Bayesian inference . Proc Int Conf on Machine Learning , p. 26293 - 26310 .

Zhang Y , Xiang T , Hospedales TM , et al. , 2018 . Deep mutual learning . IEEE/CVF Conf on Computer Vision and Pattern Recognition , p. 4320 - 4328 . 10.1109/CVPR.2018.00454 https://doi.org/10.1109/CVPR.2018.00454

Zhao Y , Li M , Lai LZ , et al. , 2022 . Federated learning with non-IID data . https://arxiv.org/abs/1806.00582 https://arxiv.org/abs/1806.00582

Views

209

Downloads

CSCD

Alert me when the article has been cited

Submit

Tools

Publicity Resources

Federated learning on non-IID and long-tailed data via dual-decoupling

Prototypical clustered federated learning for heart rate prediction

Knowledge distillation for financial large language models: a systematic review of strategies, applications, and evaluation

FedMcon: an adaptive aggregation method for federated learning via meta controller^&

Prototype-guided cross-task knowledge distillation

Related Author

Zhaohui WANG

Hongjiao LI

Jinguo LI

Renhao HU

Baojin WANG

Yongjie YIN

Hui RUAN

Yang CHEN

Related Institution

College of Computer Science and Technology, Shanghai University of Electric Power

Research Institute of Intelligent Complex Systems, Fudan University

School of Computing, Macquarie University

Department of Agricultural Sciences, University of Helsinki

Information System Department, University of Cologne

Chat

Address：Zhejiang University Press, 148 Tianmushan Road, Hangzhou, China Postal code：310028
Tel：+86-571-88273162 Email：fitee@zju.edu.cn
It is recommended to read the content of this site in Chrome&IE9+. Please switch to extreme mode in browser 360.
Cookies We use cookies to help provide and enhance our service and tailor content. By continuing, you agree to the use of cookies.

⁰