3 papers
cs.CL2026
CKV: Compressed and Composable KV Cache Reuse for Efficient LLM Inference
Chuheng Du, Junyi Chen, Hanlin Tang +7
Long-context inference is central to modern large language model (LLM) applications such as retrieval-augmented generation and multi-document reasoning. To mitigate the growing inf…
cs.LG2025
TokenFlow: Responsive LLM Text Streaming Serving under Request Burst via Preemptive Scheduling
Junyi Chen, Chuheng Du, Renyuan Liu +6
Real-time LLM interactions demand streamed token generations, where text tokens are progressively generated and delivered to users while balancing two objectives: responsiveness (i…
cs.CL2025
Pre: Enabling Deterministic Pushdown Automata for Faster Structured LLM Generation
Junyi Chen, Shihao Bai, Zaijun Wang +7
Extensive LLM applications demand efficient structured generations, particularly for LR(1) grammars, to produce outputs in specified formats (e.g., JSON). Existing methods primaril…