2 papers
cs.LG2025
KV Admission: Learning What to Write for Efficient Long-Context LLM Inference
Yen-Chieh Huang, Pi-Cheng Hsiu, Rui Fang +1
Long-context LLM inference is bottlenecked by the quadratic attention complexity and linear Key-Value (KV) cache growth. Prior approaches mitigate this via post-hoc selection or ev…
cs.DC2025
Ghidorah: Fast LLM Inference on Edge with Speculative Decoding and Hetero-Core Parallelism
Jinhui Wei, Ye Huang, Yuhui Zhou +3
In-situ LLM inference on end-user devices has gained significant interest due to its privacy benefits and reduced dependency on external infrastructure. However, as the decoding pr…