4 citations · 6 across the 2 of their papers we have counts for
4 papers
Cumulative Spatial Knowledge Distillation for Vision Transformers
Borui Zhao, Renjie Song, Jiajun Liang
Distilling knowledge from convolutional neural networks (CNNs) is a double-edged sword for vision transformers (ViTs). It boosts the performance since the image-friendly local-indu…
DOT: A Distillation-Oriented Trainer
Borui Zhao, Quan Cui, Renjie Song +1
Knowledge distillation transfers knowledge from a large model to a small one via task and distillation losses. In this paper, we observe a trade-off between task and distillation l…
Is Synthetic Data From Diffusion Models Ready for Knowledge Distillation?
Zheng Li, Yuxuan Li, Penghai Zhao +3
Diffusion models have recently achieved astonishing performance in generating high-fidelity photo-realistic images. Given their huge success, it is still unclear whether synthetic…
A Survey of Historical Learning: Learning Models with Learning History
Xiang Li, Ge Wu, Lingfeng Yang +3
New knowledge originates from the old. The various types of elements, deposited in the training history, are a large amount of wealth for improving learning deep models. In this su…