11 papers
UniPrefill: Universal Long-Context Prefill Acceleration via Block-wise Dynamic Sparsification
Qihang Fan, Huaibo Huang, Zhiying Wu +2
As large language models (LLMs) continue to advance rapidly, they are becoming increasingly capable while simultaneously demanding ever-longer context lengths. To improve the infer…
Advancing Vision Transformer with Enhanced Spatial Priors
Qihang Fan, Huaibo Huang, Mingrui Chen +2
In recent years, the Vision Transformer (ViT) has garnered significant attention within the computer vision community. However, the core component of ViT, Self-Attention, lacks exp…
FlashPrefill: Instantaneous Pattern Discovery and Thresholding for Ultra-Fast Long-Context Prefilling
Qihang Fan, Huaibo Huang, Zhiying Wu +3
Long-context modeling is a pivotal capability for Large Language Models, yet the quadratic complexity of attention remains a critical bottleneck, particularly during the compute-in…
Expand and Prune: Maximizing Trajectory Diversity for Effective GRPO in Generative Models
Shiran Ge, Chenyi Huang, Yuang Ai +3
Group Relative Policy Optimization (GRPO) is a powerful technique for aligning generative models, but its effectiveness is bottlenecked by the conflict between large group sizes an…
DiCo: Revitalizing ConvNets for Scalable and Efficient Diffusion Modeling
Yuang Ai, Qihang Fan, Xuefeng Hu +3
Diffusion Transformer (DiT), a promising diffusion model for visual generation, demonstrates impressive performance but incurs significant computational overhead. Intriguingly, ana…
Vision Transformer with Sparse Scan Prior
Yuguang Zhang, Qihang Fan, Huaibo Huang
In recent years, Transformers have achieved remarkable progress in computer vision tasks. However, their global modeling often comes with substantial computational overhead, in sta…