9 papers · 1 filter
QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction
Vincent Counathe, Ben Athiwaratkun, Christopher De Sa +1
As large language model inference shifts toward lower precision, post-training quantization (PTQ) becomes increasingly brittle, making quantization-aware training (QAT) essential f…
Extractable Memorization From First Principles
A. Feder Cooper, Marika Swanberg, Jamie Hayes +5
Recent work on extractable memorization in LLMs suffers from two contrasting validity problems. Some studies overstate extraction, e.g., relying on sequences too short to distingui…
L: Large Lookup Layers
Albert Tseng, Christopher De Sa
Modern sparse language models typically achieve sparsity through Mixture-of-Experts (MoE) layers, which dynamically route tokens to dense MLP "experts." However, dynamic hard routi…
Model-Preserving Adaptive Rounding
Albert Tseng, Zhaofeng Sun, Christopher De Sa
The goal of quantization is to produce a compressed model whose output distribution is as close to the original model's as possible. To do this tractably, most quantization algorit…
Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsity
Wentao Guo, Jikai Long, Yimeng Zeng +9
Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, the application of ZO fine-tuning in memory-…
QTIP: Quantization with Trellises and Incoherence Processing
Albert Tseng, Qingyao Sun, David Hou +1
Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing weights to low-precision datatypes. Since LLM inference is usually memory-bound, PTQ methods can…