From the 1 of 9 linked papers with an AI index.
9 papers
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Wenzong Yang, Danyang Zhang, Kun Cao +17
FastTPS is a set of techniques that reduce memory traffic and fuse operations during the token‑phase of large language model inference on AI accelerators, achieving up to a 6× spee…
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
Devleena Das, Rajeev Patwari, Vikram Kumar Bukka +3
Evaluating LLMs across many model variants -- quantized, fine-tuned, or deployment-specific -- requires running large benchmarks repeatedly, a process that can take tens of hours p…
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Inesh Chakrabarti, David Limpus, Aditi Ghai Rana +4
Context-heavy agents place unusual pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving system c…
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
Devleena Das, Rajeev Patwari, Elliott Delaye +1
Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degr…
Privatar: Scalable Privacy-preserving Multi-user VR via Secure Offloading
Jianming Tong, Hanshen Xiao, Krishna Kumar Nair +5
Multi-user virtual reality enables immersive interaction. However, rendering avatars for numerous participants on each headset incurs prohibitive computational overhead, limiting s…
KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
Sai Gokhale, Devleena Das, Rajeev Patwari +2
Long-context Large Language Models (LLMs) face significant memory bottlenecks during inference due to the linear growth of key-value (KV) cache with sequence length. While individu…