9 papers
EpiCache: Episodic KV Cache Management for Long-Term Conversation on Resource-Constrained Environments
Minsoo Kim, Arnav Kundu, Han-Byul Kim +2
Modern large language models (LLMs) extend context lengths to millions of tokens, enabling coherent, personalized responses grounded in long conversational history. However, the Ke…
Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
Mohammadreza Armandpour, Fatih Ilhan, David Harrison +6
On-policy distillation offers dense, per-token supervision for training reasoning models; however, it remains unclear under which conditions this signal is beneficial and under whi…
TIDE: Every Layer Knows the Token Beneath the Context
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +3
We revisit a universally accepted but under-examined design choice in every modern LLM: a token index is looked up once at the input embedding layer and then permanently discarded.…
SpecMD: A Comprehensive Study On Speculative Expert Prefetching
Duc Hoang, Ajay Jaiswal, Mohammad Samragh +1
Mixture-of-Experts (MoE) models enable sparse expert activation, meaning that only a subset of the model's parameters is used during each inference. However, to translate this spar…
MemoryLLM: Plug-n-Play Interpretable Feed-Forward Memory for Transformers
Ajay Jaiswal, Lauren Hannah, Han-Byul Kim +4
Understanding how transformer components operate in LLMs is important, as it is at the core of recent technological advances in artificial intelligence. In this work, we revisit th…
Mirror Speculative Decoding: Breaking the Serial Barrier in LLM Inference
Nikhil Bhendawade, Kumari Nishu, Arnav Kundu +3
Speculative decoding accelerates LLM inference by using a draft model to look ahead, but gains are capped by the cost of autoregressive draft generation: increasing draft size elev…