4 papers · 1 filter
FastTPS: An Optimized Method for LLM Token Phase for AI accelerators
Wenzong Yang, Danyang Zhang, Kun Cao +17
The popularity of large language models (LLMs) escalates an ongoing demand for effective inference. However, due to the sequential processing of tokens during the token phase in de…
MINCE: Shrinking LLM Evaluation Datasets via Few-Model Monte Carlo Calibration
Devleena Das, Rajeev Patwari, Vikram Kumar Bukka +3
Evaluating LLMs across many model variants---quantized, fine-tuned, or deployment-specific---requires running large benchmarks repeatedly, a process that can take tens of hours per…
Ablation, Statistical Inference, and Validation for KV-Cache Compression
Paolo D'Alberto, Ashish Siarasao, Elliott Delaye +1
This study systematically compares Turbo-Quant and SpectralQuant KV-cache compression, evaluating non-dominated schemes, including WHT rotation with Beta Lloyd-Max and QJL, through…
Recover-LoRA for Aggressive Quantization: Reclaiming Accuracy in 2-Bit Language Models via Low-Rank Adaptation with Knowledge Distillation on Synthetic Data
Devleena Das, Rajeev Patwari, Elliott Delaye +1
Aggressive weight quantization to 2-bit precision offers substantial throughput and memory gains for large language model (LLM) inference, but typically incurs severe accuracy degr…