Your Location:
Home >
Browse articles >
Multi-exit self-distillation with appropriate teachers
Regular Papers | Updated:2024-04-29
    • Multi-exit self-distillation with appropriate teachers

      Enhanced Publication
    • 具备合适教师的多出口自蒸馏
    • 一项关于多出口架构的研究取得重要进展,该架构通过早期停止推理来降低计算成本,适用于资源受限的环境。研究团队将多出口架构与自我蒸馏相结合,实现了在不同网络深度下同时保持高效能与良好性能的目标。然而,现有方法主要依赖深层出口或单一集成来传递知识,导致学生和教师之间的学习差距可能损害模型性能,尤其在浅层出口。为解决这一问题,研究团队提出了名为“MATE”的多出口自我蒸馏方法,为每个出口提供多样化和合适的教师知识。在MATE中,通过赋予不同权重,从所有出口中生成多个集成教师。每个出口从所有教师中获取知识,同时主要关注其主教师,以保持有效的知识传递和适当的学习差距。这种方法不仅增加了知识蒸馏的多样性,还确保了学习效率。实验结果显示,在CIFAR-100、TinyImageNet和三个细粒度数据集上,MATE在多种网络架构下均优于目前最先进的多出口自我蒸馏方法。这一研究为资源受限环境中的高效模型设计提供了新的解决方案,并为多出口架构和自我蒸馏的结合研究开辟了新方向。
    • Frontiers of Information Technology & Electronic Engineering   Vol. 25, Issue 4, Pages: 585-599(2024)
    • DOI:10.1631/FITEE.2200644    

      CLC: TP181
    • Published:0 April 2024

      Received:16 December 2022

      Accepted:2023-07-04

    Scan QR Code

  • WUJIE SUN, DEFANG CHEN, CAN WANG, et al. Multi-exit self-distillation with appropriate teachers. [J]. Frontiers of information technology & electronic engineering, 2024, 25(4): 585-599. DOI: 10.1631/FITEE.2200644.

  •  
  •  

0

Views

111

Downloads

0

CSCD

>
Alert me when the article has been cited
Submit
Tools
Download
Export Citation
Share
Add to favorites
Add to my album

Related Articles

Federated learning on non-IID and long-tailed data via dual-decoupling
Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives
Forget less, count better: a domain-incremental self-distillation learning benchmark for lifelong crowd counting
Binary neural networks for speech recognition

Related Author

Baojin WANG
Renhao HU
Jinguo LI
Hongjiao LI
Zhaohui WANG
Tao SHEN
Jie ZHANG
Xinkang JIA

Related Institution

College of Computer Science and Technology, Shanghai University of Electric Power
School of Public Affairs, Zhejiang University
School of Software Technology, Zhejiang University
School of Information Science and Technology, Xiamen University
State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences
0