6 papers
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they req…
Attention Transfer Is Not Universally Effective for Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4
A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard stud…
Beyond Instance Consistency: Investigating View Diversity in Self-supervised Learning
Huaiyuan Qin, Muli Yang, Siyuan Hu +4
Self-supervised learning (SSL) conventionally relies on the instance consistency paradigm, assuming that different views of the same image can be treated as positive pairs. However…
Hybrid Data-Free Knowledge Distillation
Jialiang Tang, Shuo Chen, Chen Gong
Data-free knowledge distillation aims to learn a compact student network from a pre-trained large teacher network without using the original training data of the teacher network. E…
Direct Distillation between Different Domains
Jialiang Tang, Shuo Chen, Gang Niu +4
Knowledge Distillation (KD) aims to learn a compact student network using knowledge from a large pre-trained teacher network, where both networks are trained on data from the same…
A Survey on Multi-output Learning
Donna Xu, Yaxin Shi, Ivor W. Tsang +3
Multi-output learning aims to simultaneously predict multiple outputs given an input. It is an important learning problem due to the pressing need for sophisticated decision making…