5 papers
SPARE: Structural Parameter-Free Affinity Regularization for Flow Matching
Zong-Wei Hong, Jinglun Li, Shen Zhang +3
Denoising diffusion transformers achieve strong generation quality but converge slowly during training. Regularizing their internal representations has emerged as an effective acce…
The Velocity Deficit: Initial Energy Injection for Flow Matching
Linze Li, Zong-Wei Hong, Shen Zhang +4
While Flow Matching theoretically guarantees constant-velocity trajectories, we identify a critical breakdown in high-dimensional practice: the Velocity Deficit. We show that the M…
VeCoR -- Velocity Contrastive Regularization for Flow Matching
Zong-Wei Hong, Jing-lun Li, Lin-Ze Li +2
Flow Matching (FM) has recently emerged as a principled and efficient alternative to diffusion models. Standard FM encourages the learned velocity field to follow a target directio…
LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
Shen Zhang, Siyuan Liang, Yaning Tan +9
Diffusion transformers (DiTs) struggle to generate images at resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encodings(PE),…
Optimizing Knowledge Distillation in Transformers: Enabling Multi-Head Attention without Alignment Barriers
Zhaodong Bing, Linze Li, Jiajun Liang
Knowledge distillation (KD) in transformers often faces challenges due to misalignment in the number of attention heads between teacher and student models. Existing methods either…