5 papers
Anti-Shortcut Distillation via Temporal Negative Knowledge Transfer
Syed Muhammad Raza, Omer Tariq, Jeongbae Son
Knowledge distillation (KD) trains a compact student by attracting it towards a converged teacher. It is silent about which directions the teacher itself learned to suppress: repul…
Latent-Condensed Transformer for Efficient Long Context Modeling
Zeng You, Yaofo Chen, Qiuwu Chen +5
Large language models (LLMs) face significant challenges in processing long contexts due to the linear growth of the key-value (KV) cache and quadratic complexity of self-attention…
ProCache: Constraint-Aware Feature Caching with Selective Computation for Diffusion Transformer Acceleration
Fanpu Cao, Yaofo Chen, Zeng You +1
Diffusion Transformers (DiTs) have achieved state-of-the-art performance in generative modeling, yet their high computational cost hinders real-time deployment. While feature cachi…
Training-free Context-adaptive Attention for Efficient Long Context Modeling
Zeng You, Yaofo Chen, Shuhai Zhang +5
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks. These capabilities stem primarily from the self-att…
Core Context Aware Transformers for Long Context Language Modeling
Yaofo Chen, Zeng You, Shuhai Zhang +4
Transformer-based Large Language Models (LLMs) have exhibited remarkable success in extensive tasks primarily attributed to self-attention mechanism, which requires a token to cons…