5 papers
Copy the Same, Distill the Difference: Initializing Linear Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +9
Linear Vision Transformers (ViTs) are designed to replace the attention in Softmax ViTs with the linear-complexity attention operator for more efficient token routing, but they req…
DiD It in 87 Minutes: A Label-Free Softmax-to-Linear Adaptation of Vision Transformers for Object Detection
Huaiyuan Qin, Gabriel James Goenawan, Zihang Lin +2
While linear attention is a compelling mechanism for high-resolution object detection due to its reduced cost for global token mixing, converting the Softmax-attention ViT backbone…
Attention Transfer Is Not Universally Effective for Vision Transformers
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +4
A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard stud…
Beyond Loss Values: Robust Dynamic Pruning via Loss Trajectory Alignment
Huaiyuan Qin, Muli Yang, Gabriel James Goenawan +5
Existing dynamic data pruning methods often fail under noisy-label settings, as they typically rely on per-sample loss as the ranking criterion. This could mistakenly lead to prese…
Your AI-Generated Image Detector Can Secretly Achieve SOTA Accuracy, If Calibrated
Muli Yang, Gabriel James Goenawan, Henan Wang +7
Despite being trained on balanced datasets, existing AI-generated image detectors often exhibit systematic bias at test time, frequently misclassifying fake images as real. We hypo…