Your Location:
Home >
Browse articles >
Multi-exit self-distillation with appropriate teachers
Regular Papers | Updated:2024-04-29
    • Multi-exit self-distillation with appropriate teachers

    • 具备合适教师的多出口自蒸馏
    • “一项关于多出口架构的研究取得重要进展,该架构通过早期停止推理来降低计算成本,适用于资源受限的环境。研究团队将多出口架构与自我蒸馏相结合,实现了在不同网络深度下同时保持高效能与良好性能的目标。然而,现有方法主要依赖深层出口或单一集成来传递知识,导致学生和教师之间的学习差距可能损害模型性能,尤其在浅层出口。为解决这一问题,研究团队提出了名为“MATE”的多出口自我蒸馏方法,为每个出口提供多样化和合适的教师知识。在MATE中,通过赋予不同权重,从所有出口中生成多个集成教师。每个出口从所有教师中获取知识,同时主要关注其主教师,以保持有效的知识传递和适当的学习差距。这种方法不仅增加了知识蒸馏的多样性,还确保了学习效率。实验结果显示,在CIFAR-100、TinyImageNet和三个细粒度数据集上,MATE在多种网络架构下均优于目前最先进的多出口自我蒸馏方法。这一研究为资源受限环境中的高效模型设计提供了新的解决方案,并为多出口架构和自我蒸馏的结合研究开辟了新方向。”
    • Frontiers of Information Technology & Electronic Engineering   Vol. 25, Issue 4, Pages: 585-599(2024)
    • DOI:10.1631/FITEE.2200644    

      CLC: TP181
    • Received:16 December 2022,

      Accepted:04 July 2023,

      Published:0 April 2024

    Scan QR Code

  • Wujie SUN, Defang CHEN, Can WANG, et al. Multi-exit self-distillation with appropriate teachers[J]. Frontiers of Information Technology & Electronic Engineering, 2024, 25(4): 585-599. DOI: 10.1631/FITEE.2200644.

  •  
  •  

0

Views

304

Downloads

0

CSCD

>
Alert me when the article has been cited
Submit
Tools
Download
Export Citation
Share
Add to favorites
Add to my album

Related Articles

Knowledge distillation for financial large language models: a systematic review of strategies, applications, and evaluation
Prototype-guided cross-task knowledge distillation
Federated learning on non-IID and long-tailed data via dual-decoupling
Federated mutual learning: a collaborative machine learning method for heterogeneous data, models, and objectives
Forget less, count better: a domain-incremental self-distillation learning benchmark for lifelong crowd counting

Related Author

Jianzong WANG
Junfei XIE
Xiaoyang QU
Xulong ZHANG
Jiaqi SHI
Yahong HAN
Aming WU
Peng LI

Related Institution

Institute of Advanced Technology, University of Science and Technology of China
Ping An Technology (Shenzhen) Co., Ltd.
School of Electronic Engineering, Xidian University
Songshan Laboratory
College of Intelligence and Computing, Tianjin University
0