FOLLOWUS
1.College of Computer Science and Technology, Zhejiang University, Hangzhou 310027, China
2.School of Software Technology, Zhejiang University, Hangzhou 310027, China
3.School of Public Affairs, Zhejiang University, Hangzhou 310027, China
‡Corresponding author
纸质出版日期:2023-10-0 ,
网络出版日期:2023-08-05,
收稿日期:2023-02-20,
录用日期:2023-04-07
Scan QR Code
沈弢, 张杰, 贾鑫康, 等. 联邦相互学习:一种针对异构数据、模型和目标的协同机器学习方法[J]. 信息与电子工程前沿(英文), 2023,24(10):1390-1402.
TAO SHEN, JIE ZHANG, XINKANG JIA, et al. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives. [J]. Frontiers of information technology & electronic engineering, 2023, 24(10): 1390-1402.
沈弢, 张杰, 贾鑫康, 等. 联邦相互学习:一种针对异构数据、模型和目标的协同机器学习方法[J]. 信息与电子工程前沿(英文), 2023,24(10):1390-1402. DOI: 10.1631/FITEE.2300098.
TAO SHEN, JIE ZHANG, XINKANG JIA, et al. Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives. [J]. Frontiers of information technology & electronic engineering, 2023, 24(10): 1390-1402. DOI: 10.1631/FITEE.2300098.
联邦学习(FL)是深度学习中的一种新技术,可以让客户端在保留各自隐私数据的情况下协同训练模型。然而,由于每个客户端的数据分布、算力和场景都不同,联邦学习面临客户端异构环境的挑战。现有方法(如FedAvg)无法有效满足每个客户的定制化需求。为解决联邦学习中的异构挑战,本文首先详述了数据、模型和目标(DMO)这3个主要异构来源,然后提出一种新的联邦相互学习(FML)框架。该框架使得每个客户端都能训练一个考虑到数据异构(DH)的个性化模型。在模型异构(MH)问题上,引入一种“模因模型”作为个性化模型与全局模型之间的中介,并且采用深度相互学习(DML)的知识蒸馏技术在两个异构模型之间传递知识。针对目标异构(OH)问题,通过共享部分模型参数,设计针对特定任务的个性化模型,同时,利用模因模型进行相互学习。本研究通过实验评估了FML在应对DMO异构性方面的表现,并与其他常见FL方法在相似场景下进行对比。实验结果表明,FML在处理FL环境中的DMO问题的表现卓越,优于其他方法。
Federated learning (FL) is a novel technique in deep learning that enables clients to collaboratively train a shared model while retaining their decentralized data. However
researchers working on FL face several unique challenges
especially in the context of heterogeneity. Heterogeneity in data distributions
computational capabilities
and scenarios among clients necessitates the development of customized models and objectives in FL. Unfortunately
existing works such as FedAvg may not effectively accommodate the specific needs of each client. To address the challenges arising from heterogeneity in FL
we provide an overview of the heterogeneities in data
model
and objective (DMO). Furthermore
we propose a novel framework called federated mutual learning (FML)
which enables each client to train a personalized model that accounts for the data heterogeneity (DH). A "meme model" serves as an intermediary between the personalized and global models to address model heterogeneity (MH). We introduce a knowledge distillation technique called deep mutual learning (DML) to transfer knowledge between these two models on local data. To overcome objective heterogeneity (OH)
we design a shared global model that includes only certain parts
and the personalized model is task-specific and enhanced through mutual learning with the meme model. We evaluate the performance of FML in addressing DMO heterogeneities through experiments and compare it with other commonly used FL methods in similar scenarios. The results demonstrate that FML outperforms other methods and effectively addresses the DMO challenges encountered in the FL setting.
联邦学习知识蒸馏隐私保护异构环境
Federated learningKnowledge distillationPrivacy preservingHeterogeneous environment
Alam S, Liu LY, Yan M, et al., 2023. FedRolex: model-heterogeneous federated learning with rolling sub-model extraction. https://arxiv.org/abs/2212.01548https://arxiv.org/abs/2212.01548
Chen HT, Wang YH, Xu C, et al., 2019. Data-free learning of student networks. IEEE/CVF Int Conf on Computer Vision, p.3513-3521. 10.1109/ICCV.2019.00361https://doi.org/10.1109/ICCV.2019.00361
Chen HY, Chao WL, 2022. On bridging generic and personalized federated learning for image classification. https://arxiv.org/abs/2107.00778https://arxiv.org/abs/2107.00778
Corchado JM, Li WG, Bajo J, et al., 2016. Special issue on distributed computing and artificial intelligence. Front Inform Technol Electron Eng, 17(4):281-282. 10.1631/FITEE.DCAI2015https://doi.org/10.1631/FITEE.DCAI2015
Gao DS, Ju C, Wei XG, et al., 2020. HHHFL: hierarchical heterogeneous horizontal federated learning for electroencephalography. https://arxiv.org/abs/1909.05784https://arxiv.org/abs/1909.05784
Gao JQ, Li JQ, Shan HM, et al., 2023. Forget less, count better: a domain-incremental self-distillation learning benchmark for lifelong crowd counting. Front Inform Technol Electron Eng, 24(2):187-202. 10.1631/FITEE.2200380https://doi.org/10.1631/FITEE.2200380
He CY, Annavaram M, Avestimehr S, et al., 2021. FedNAS: federated deep learning via neural architecture search. https://arxiv.org/abs/2004.08546v1https://arxiv.org/abs/2004.08546v1
Hinton G, Vinyals O, Dean J, 2015. Distilling the knowledge in a neural network. https://arxiv.org/abs/1503.02531https://arxiv.org/abs/1503.02531
Jiang YH, Konečný J, Rush K, et al., 2023. Improving federated learning personalization via model agnostic meta learning. https://arxiv.org/abs/1909.12488https://arxiv.org/abs/1909.12488
Kairouz P, McMahan HB, Avent B, et al., 2021. Advances and open problems in federated learning. Found Trends Mach Learn, 14(1-2):1-210. 10.1561/2200000083https://doi.org/10.1561/2200000083
Khodak M, Balcan MF, Talwalkar A, 2019. Adaptive gradient-based meta-learning methods. https://arxiv.org/abs/1906.02717https://arxiv.org/abs/1906.02717
Krizhevsky A, 2009. Learning Multiple Layers of Features from Tiny Images. Master Thesis, Department of Computer Science, University of Toronto, Canada.
LeCun Y, Boser B, Denker J, et al., 1989. Handwritten digit recognition with a back-propagation network. Proc 2nd Int Conf on Neural Information Processing Systems, p.396-404.
LeCun Y, Bottou L, Bengio Y, et al., 1998. Gradient-based learning applied to document recognition. Proc IEEE, 86(11):2278-2324. 10.1109/5.726791https://doi.org/10.1109/5.726791
Li DL, Wang JP, 2019. FedMD: heterogenous federated learning via model distillation. https://arxiv.org/abs/1910.03581https://arxiv.org/abs/1910.03581
Li JH, 2018. Cyber security meets artificial intelligence: a survey. Front Inform Technol Electron Eng, 19(12):1462-1474. 10.1631/FITEE.1800573https://doi.org/10.1631/FITEE.1800573
Li T, Sahu AK, Zaheer M, et al., 2020. Federated optimization in heterogeneous networks. https://arxiv.org/abs/1812.06127v5https://arxiv.org/abs/1812.06127v5
Li WH, Bilen H, 2020. Knowledge distillation for multi-task learning. Proc European Conf on Computer Vision, p.163-176. 10.1007/978-3-030-65414-6_13https://doi.org/10.1007/978-3-030-65414-6_13
Li X, Huang KX, Yang WH, et al., 2019. On the convergence of FedAvg on non-IID data. https://arxiv.org/abs/1907.02189https://arxiv.org/abs/1907.02189
Li X, Yang WH, Wang SS, et al., 2021. Communication efficient decentralized training with multiple local updates. https://arxiv.org/abs/1910.09126v1https://arxiv.org/abs/1910.09126v1
Lian XR, Zhang C, Zhang H, et al., 2017. Can decentralized algorithms outperform centralized algorithms? A case study for decentralized parallel stochastic gradient descent. Proc 31st Int Conf on Neural Information Processing Systems, p.5336-5346.
Liang PP, Liu T, Liu ZY, et al., 2020. Think locally, act globally: federated learning with local and global representations. https://arxiv.org/abs/2001.01523https://arxiv.org/abs/2001.01523
Lim WYB, Luong NC, Hoang DT, et al., 2020Federated learning in mobile edge networks: a comprehensive survey. IEEE Commun Surv Tutor, 22(3):2031-2063. 10.1109/COMST.2020.2986024https://doi.org/10.1109/COMST.2020.2986024
Liu FL, Wu X, Ge S, et al., 2020. Federated learning for vision-and-language grounding problems. Proc AAAI Conf Artif Intell, 34(7):11572-11579. 10.1609/aaai.v34i07.6824https://doi.org/10.1609/aaai.v34i07.6824
Liu PX, Jiang JM, Zhu GX, et al., 2022. Training time minimization for federated edge learning with optimized gradient quantization and bandwidth allocation. Front Inform Technol Electron Eng, 23(8):1247-1263. 10.1631/FITEE.2100538https://doi.org/10.1631/FITEE.2100538
McMahan B, Moore E, Ramage D, et al., 2017. Communication-efficient learning of deep networks from decentralized data. Proc 20th Int Conf on Artificial Intelligence and Statistics, p.1273-1282.
Padhya M, Jinwala DC, 2019. MULKASE: a novel approach for key-aggregate searchable encryption for multi-owner data. Front Inform Technol Electron Eng, 20(12):1717-1748. 10.1631/FITEE.1800192https://doi.org/10.1631/FITEE.1800192
Pan YH, 2017. Special issue on artificial intelligence 2.0. Front Inform Technol Electron Eng, 18(1):1-2. 10.1631/FITEE.1710000https://doi.org/10.1631/FITEE.1710000
Pan YH, 2018. 2018 special issue on artificial intelligence 2.0: theories and applications. Front Inform Technol Electron Eng, 19(1):1-2. 10.1631/FITEE.1810000https://doi.org/10.1631/FITEE.1810000
Smith V, Chiang CK, Sanjabi M, et al., 2017. Federated multi-task learning. Proc 31st Int Conf on Neural Information Processing Systems, p.4427-4437.
Wang J, Li R, Wang J, et al., 2020. Artificial intelligence and wireless communications. Front Inform Technol Electron Eng, 21(10):1413-1425. 10.1631/FITEE.1900527https://doi.org/10.1631/FITEE.1900527
Wang TZ, Zhu JY, Torralba A, et al., 2020. Dataset distillation. https://arxiv.org/abs/1811.10959https://arxiv.org/abs/1811.10959
Wu BC, Dai XL, Zhang PZ, et al., 2019. FBNet: hardware-aware efficient ConvNet design via differentiable neural architecture search. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10726-10734. 10.1109/CVPR.2019.01099https://doi.org/10.1109/CVPR.2019.01099
Wu JX, Li JH, Ji XS, 2018. Security for cyberspace: challenges and opportunities. Front Inform Technol Electron Eng, 19(12):1459-1461. 10.1631/FITEE.1840000https://doi.org/10.1631/FITEE.1840000
Yang Q, Liu Y, Cheng Y, et al., 2019. Federated Learning. Springer, Cham, Switzerland, p.1-207.
Yu T, Bagdasaryan E, Shmatikov V, 2022. Salvaging federated learning by local adaptation. https://arxiv.org/abs/2002.04758https://arxiv.org/abs/2002.04758
Zhang X, Li YC, Li WP, et al., 2022. Personalized federated learning via variational Bayesian inference. Proc Int Conf on Machine Learning, p.26293-26310.
Zhang Y, Xiang T, Hospedales TM, et al., 2018. Deep mutual learning. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.4320-4328. 10.1109/CVPR.2018.00454https://doi.org/10.1109/CVPR.2018.00454
Zhao Y, Li M, Lai LZ, et al., 2022. Federated learning with non-IID data. https://arxiv.org/abs/1806.00582https://arxiv.org/abs/1806.00582
关联资源
相关文章
相关作者
相关机构