4 papers · 1 filter
Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Level
Nan Jia, Haojin Yang, Xing Ma +6
On-policy distillation (OPD) trains a student on its own trajectories with token-level teacher feedback and often outperforms off-policy distillation and standard reinforcement lea…
CuBridge: An LLM-Based Framework for Understanding and Reconstructing High-Performance Attention Kernels
Xing Ma, Yangjie Zhou, Wu Sun +6
Efficient CUDA implementations of attention mechanisms are critical to modern deep learning systems, yet supporting diverse and evolving attention variants remains challenging. Exi…
ELSA: Exact Linear-Scan Attention for Fast and Memory-Light Vision Transformers
Chih-Chung Hsu, Xin-Di Ma, Wo-Ting Liao +1
Existing attention accelerators often trade exact softmax semantics, depend on fused Tensor Core kernels, or incur sequential depth that limits FP32 throughput on long sequences. W…
Structural Pruning of Pre-trained Language Models via Neural Architecture Search
Aaron Klein, Jacek Golebiowski, Xingchen Ma +2
Pre-trained language models (PLM), for example BERT or RoBERTa, mark the state-of-the-art for natural language understanding task when fine-tuned on labeled data. However, their la…