12 papers
LeapQuant: Efficient Linear Attention with Accurate Recurrent State Quantization
Yi Pan, Haocheng Xi, Kan Zhu +10
Recent LLMs increasingly adopt hybrid designs that replace standard attention with linear attention, such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA). Although they comp…
HERALD: High-Throughput Block Diffusion LLM Serving via CPU-GPU Cooperative KV Cache Retrieval
Omin Kwon, Doyeon Kim, Jongseok Park +3
The KV cache dominates GPU memory in long-context LLM serving, crowding out batch capacity and leaving GPU compute idle. Offloading the cache to CPU DRAM restores capacity, but the…
Uncovering Intra-expert Activation Sparsity for Efficient Mixture-of-Expert Model Execution
Jongseok Park, Sunga Kim, Zhenyu Gu +2
Mixture of Experts (MoE) architecture has become the standard for state-of-the-art large language models, owing to its computational efficiency through sparse expert activation. Ho…
Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
Yilong Zhao, Xiaonan Nie, Kan Zhu +6
Context parallelism (CP) has been widely adopted to support the growing context length in foundation model pretraining. However, existing designs fail to handle the large variation…
Qrita: High-performance Top-k and Top-p using Pivot-based Truncation and Selection
Jongseok Park, Sunga Kim, Alvin Cheung +1
Despite their importance in model sampling, efficient implementation of Top-k and Top-p algorithms for large vocabularies remains a significant challenge. Existing approaches often…
Speculative Decoding: Performance or Illusion?
Xiaoxuan Liu, Jiaxiang Yu, Jongseok Park +2
Speculative decoding (SD) has become a popular technique to accelerate Large Language Model (LLM) inference, yet its real-world effectiveness remains unclear as prior evaluations r…
Accelerating Large-Scale Reasoning Model Inference with Sparse Self-Speculative Decoding
Yilong Zhao, Jiaming Tang, Kan Zhu +10
Reasoning language models have demonstrated remarkable capabilities on challenging tasks by generating elaborate chain-of-thought (CoT) solutions. However, such lengthy generation…
RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
Jacky Kwok, Christopher Agia, Rohan Sinha +5
Vision-Language-Action (VLA) models have demonstrated remarkable capabilities in visuomotor control, yet ensuring their robustness in unstructured real-world environments remains a…
Radial Attention: Sparse Attention with Energy Decay for Long Video Generation
Xingyang Li, Muyang Li, Tianle Cai +11
Recent advances in diffusion models have enabled high-quality video generation, but the additional temporal dimension significantly increases computational costs, making training a…
: Faster Test-Time Scaling through Speculative Drafts
Mert Cemri, Nived Rajaraman, Rishabh Tiwari +6
Scaling test-time compute has driven the recent advances in the reasoning capabilities of large language models (LLMs), typically by allocating additional computation for more thor…
PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applications
Kuntai Du, Bowen Wang, Chen Zhang +9
Besides typical generative applications, like ChatGPT, GitHub Copilot, and Cursor, we observe an emerging trend that LLMs are increasingly used in traditional discriminative tasks,…
The Dawn of Disaggregation and the Coherence Conundrum: A Call for Federated Coherence
Jaewan Hong, Marcos K. Aguilera, Emmanuel Amaro +3
Disaggregated memory is an upcoming data center technology that will allow nodes (servers) to share data efficiently. Sharing data creates a debate on the level of cache coherence…