101 citations · 102 across the 7 of their papers we have counts for
8 papers
AbridgeKV: Compressing Long Reasoning into the KV Cache of Learned Pause Tokens
Jack Cai, Lijie Yang, Vasilis Kontonis +2
Long chain-of-thought decoding is memory-bound: the KV cache of the generated tokens grows linearly with the test-time compute being spent. Evicting cache entries does not fix this…
Thought-Level Beam Search for Reasoning
Lijie Yang, Hongyin Luo, Jiawei Zhao +2
Test-time compute scaling is a primary driver of performance in large reasoning models (LRMs), but extreme inefficiency bounds current approaches, shifting the critical question fr…
Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel
Hongyi Jin, Bohan Hou, Guanjie Wang +18
Modern GPU workloads, especially large language model (LLM) inference, suffer from kernel launch overheads and coarse synchronization that limit inter-kernel parallelism. Recent me…
Axe: A Simple Unified Layout Abstraction for Machine Learning Compilers
Bohan Hou, Hongyi Jin, Guanjie Wang +7
Scaling modern deep learning workloads demands coordinated placement of data and compute across device meshes, memory hierarchies, and heterogeneous accelerators. We present Axe La…
CaveAgent: Transforming LLMs into Stateful Runtime Operators
Maohao Ran, Zhenglin Wan, Cooper Lin +21
LLM-based agents are increasingly capable of complex task execution, yet current agentic systems remain constrained by text-centric paradigms that struggle with long-horizon tasks…
TidalDecode: Fast and Accurate LLM Decoding with Position Persistent Sparse Attention
Lijie Yang, Zhihao Zhang, Zhuofu Chen +2
Large language models (LLMs) have driven significant advancements across diverse NLP tasks, with long-context models gaining prominence for handling extended inputs. However, the e…
Accelerating Retrieval-Augmented Language Model Serving with Speculation
Zhihao Zhang, Alan Zhu, Lijie Yang +4
Retrieval-augmented language models (RaLM) have demonstrated the potential to solve knowledge-intensive natural language processing (NLP) tasks by combining a non-parametric knowle…
SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification
Xupeng Miao, Gabriele Oliaro, Zhihao Zhang +12
This paper introduces SpecInfer, a system that accelerates generative large language model (LLM) serving with tree-based speculative inference and verification. The key idea behind…