20 citations · 32 across the 11 of their papers we have counts for
13 papers
The Art of Closed-Formula Defaults: Search-Free Code Generation for Tensor Operators
Paolo D'Alberto, Ashish Sirasao
Agentic search and automated optimization of GPU kernels are powerful tools for large language model inference. Their effectiveness, however, depends not on the sophistication of t…
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Wenzong Yang, Danyang Zhang, Kun Cao +17
The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in de…
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
Devleena Das, Rajeev Patwari, Vikram Kumar Bukka +3
Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per…
UltraQuant: 4-bit KV Caching for Context-Heavy Agents
Inesh Chakrabarti, David Limpus, Aditi Ghai Rana +4
Context-heavy agents place substantial pressure on the key-value (KV) cache: long prefixes are reused across many short turns, while concurrency determines whether the serving syst…
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
Devleena Das, Rajeev Patwari, Elliott Delaye +1
Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degr…
Privatar: Scalable Privacy-preserving Multi-user VR via Secure Offloading
Jianming Tong, Hanshen Xiao, Krishna Kumar Nair +5
Multi-user virtual reality enables immersive interaction. However, rendering avatars for numerous participants on each headset incurs prohibitive computational overhead, limiting s…
KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
Sai Gokhale, Devleena Das, Rajeev Patwari +2
Long-context Large Language Models (LLMs) face significant memory bottlenecks during inference due to the linear growth of key-value (KV) cache with sequence length. While individu…
Recover-LoRA: Data-Free Accuracy Recovery of Degraded Language Models via Low-Rank Adaptation
Devleena Das, Rajeev Patwari, Ashish Sirasao
Inference optimizations such as quantization, pruning, format and datatype conversion, model export, and serialization can lead to functional degradations in language model task pe…
Forecasting LLM Inference Performance via Hardware-Agnostic Analytical Modeling
Rajeev Patwari, Ashish Sirasao, Devleena Das
Large language models (LLMs) have been increasingly deployed as local agents on personal devices with CPUs, NPUs and integrated GPUs. However, forecasting inference performance on…
Exploring the Dynamic Scheduling Space of Real-Time Generative AI Applications on Emerging Heterogeneous Systems
Rachid Karami, Rajeev Patwari, Hyoukjun Kwon +1
The integration of generative AI models, particularly large language models (LLMs), into real-time multi-model AI applications such as video conferencing and gaming is giving rise…
XRBench: An Extended Reality (XR) Machine Learning Benchmark Suite for the Metaverse
Hyoukjun Kwon, Krishnakumar Nair, Jamin Seo +15
Real-time multi-task multi-model (MTMM) workloads, a new form of deep learning inference workloads, are emerging for applications areas like extended reality (XR) to support metave…
MLPerf Inference Benchmark
Vijay Janapa Reddi, Christine Cheng, David Kanter +44
Machine-learning (ML) hardware and software system demand is burgeoning. Driven by ML applications, the number of different ML inference systems has exploded. Over 100 organization…
Quantizing Convolutional Neural Networks for Low-Power High-Throughput Inference Engines
Sean O. Settle, Manasa Bollavaram, Paolo D'Alberto +6
Deep learning as a means to inferencing has proliferated thanks to its versatility and ability to approach or exceed human-level accuracy. These computational models have seemingly…