Showing 2026Show all
2 papers · 1 filter
cs.CV2026
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they req…
cs.CV2026
Attention Transfer Is Not Universally Effective for Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4
A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard stud…