Distilling Drifting Transformers with Representation Autoencoders

Jiawei Zhang, Mengfei Xia, Gen Li +1

Despite the significant training acceleration and promising performance, Representation Autoencoders (RAEs) are mainly criticized for poor distillation effectiveness. In this work,…