28 papers
REFACT: Adaptive Fact Restatement for Compact and Faithful Chain-of-Thought Reasoning
Zhensheng Jin, Xin Dai, Zhenghao Liu +5
Large Language Models (LLMs) increasingly leverage long-form reasoning to solve complex tasks, yet their reasoning processes can deviate from the provided context when evidence is…
Beyond SFT-to-RL: Pre-alignment via Black-Box On-Policy Distillation for Multimodal RL
Sudong Wang, Weiquan Huang, Xiaomin Yu +9
The standard post-training recipe for large multimodal models (LMMs) applies supervised fine-tuning (SFT) on curated demonstrations followed by reinforcement learning with verifiab…
Beyond Reward Engineering: A Data Recipe for Long-Context Reinforcement Learning
Xiaoyue Xu, Sikui Zhang, Xiaorong Wang +2
Long-context reasoning is an essential capability for large language models, particularly when they are deployed as autonomous agents that must reason over lengthy trajectories. Re…
Rethinking the Role of Efficient Attention in Hybrid Architectures
Ziqing Qiao, Yinuo Xu, Chaojun Xiao +6
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent se…
APB-V: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention
Yuxiang Huang, Mingye Li, Xu Han +7
The efficiency of long-video inference remains a critical bottleneck, mainly due to the dense computation in the prefill stage of Large Multimodal Models (LMMs). Existing methods e…
DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
Chenyang Song, Weilin Zhao, Xu Han +3
While Mixture-of-Experts (MoE) scales model capacity without proportionally increasing computation, its massive total parameter footprint creates significant storage and memory-acc…