4 papers · 2 filters
Characterizing High Bandwidth Flash for LLM Serving
Zack Yu, Chloe Wong, Coleman Hooper +7
Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth…
AsyncOPD: How Stale Can On-Policy Distillation Be?
Wonjun Kang, Kevin Galim, Seunghyuk Oh +9
On-policy distillation (OPD) trains a student on its own rollouts guided by teacher feedback and is becoming increasingly important for large language model (LLM) post-training. Li…
EfficientRollout: System-Aware Self-Speculative Decoding for RL Rollouts
Minseo Kim, Minjae Lee, Seunghyuk Oh +7
Reinforcement learning (RL) has become a representative post-training paradigm for large language models (LLMs), enabling strong reasoning and agentic capabilities. However, rollou…
TABED: Test-Time Adaptive Ensemble Drafting for Robust Speculative Decoding in LVLMs
Minjae Lee, Wonjun Kang, Byeongkeun Ahn +6
Speculative decoding (SD) has proven effective for accelerating LLM inference by quickly generating draft tokens and verifying them in parallel. However, SD remains largely unexplo…