3 papers
cs.LG2025
Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost
Haojun Xia, Xiaoxia Wu, Jisen Li +12
The KV cache is a dominant memory bottleneck for LLM inference. While 4-bit KV quantization preserves accuracy, 2-bit often degrades it, especially on long-context reasoning. We cl…
cs.AI2024
Alopex: A Computational Framework for Enabling On-Device Function Calls with LLMs
Yide Ran, Zhaozhuo Xu, Yuhang Yao +9
The rapid advancement of Large Language Models (LLMs) has led to their increased integration into mobile devices for personalized assistance, which enables LLMs to call external AP…
cs.AI2024
TensorOpera Router: A Multi-Model Router for Efficient LLM Inference
Dimitris Stripelis, Zijian Hu, Jipeng Zhang +6
With the rapid growth of Large Language Models (LLMs) across various domains, numerous new LLMs have emerged, each possessing domain-specific expertise. This proliferation has high…