14 papers
Output-Aware Rotation for INT2 KV-Cache Quantization
Vincent-Daniel Yun, Woosang Lim, Minsoo Cheong +4
The key-value (KV) cache has become a major memory and bandwidth bottleneck in long-context large language model inference, making ultra-low-bit quantization increasingly important…
OTRO: Oblivious Tokenization Path with Square-Root ORAM
Jonghyun Lee, Yongqin Wang, Rachit Rajat +3
The CPU-side large language model (LLM) tokenizer is a critical security gap in LLM serving through a confidential computing stack with CPU and GPU trusted execution environments (…
DuetServe: Harmonizing Prefill and Decode for LLM Serving via Adaptive GPU Multiplexing
Lei Gao, Chaoyi Jiang, Hossein Entezari Zarch +3
Modern LLM serving systems must sustain high throughput while meeting strict latency SLOs across two distinct inference phases: compute-intensive prefill and memory-bound decode ph…
Infrastructure for Valuable, Tradable, and Verifiable Agent Memory
Mengyuan Li, Lei Gao, Haoxuan Xu +5
Every API token you spend is your accumulated wealth; once you can prove its value and the effort behind it, you can resell it. As autonomous agents repeatedly call models and tool…
LRD-MPC: Efficient MPC Inference through Low-rank Decomposition
Tingting Tang, Yongqin Wang, Murali Annavaram
Secure Multi-party Computation (MPC) enables untrusted parties to jointly compute a function without revealing their inputs. Its application to machine learning (ML) has gained sig…
Differentially Private Retrieval-Augmented Generation
Tingting Tang, James Flemings, Yongqin Wang +1
Retrieval-augmented generation (RAG) is a widely used framework for reducing hallucinations in large language models (LLMs) on domain-specific tasks by retrieving relevant document…