From the 1 of 13 linked papers with an AI index.
13 papers
GPTQ-2D: Cubic-Time Two-Sided Adaptive Rounding
Jiale Chen, Torsten Hoefler, Dan Alistarh
The paper introduces GPTQ-2D, an algorithm that rounds a real matrix to integers under a two-sided quadratic metric in cubic time by processing entries anti-diagonal by anti-diagon…
Can AI Weather Models Predict Beyond Two Weeks? A Quantitative Benchmark and Analysis of Long Rollouts
Fanny Lehmann, Firat Ozdemir, Yun Cheng +4
While AI weather models excel at short-to-medium range forecasts (up to 15 days), they frequently suffer from ill-defined "instabilities" when rolled out over longer horizons. This…
When Data Is Scarce: Scaling Sparse Language Models with Repeated Training
Boqian Wu, Qiao Xiao, Patrik Okanovic +6
Scaling laws for dense LLMs under infinite data are well explored, but how sparsity interacts with limited data is not. In this work, we study sparse training in data-constrained r…
Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
Qiao Xiao, Boqian Wu, Patrik Okanovic +6
Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model…
WUSH: Near-Optimal Adaptive Transforms for LLM Quantization
Jiale Chen, Vage Egiazarian, Roberto L. Castro +2
Quantizing LLM weights and activations is a standard approach for efficient deployment, but a few extreme outliers can stretch the dynamic range and amplify low-bit quantization er…
The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
Jiale Chen, Yalda Shabanzadeh, Elvir CrnÄeviÄ +2
Quantizing the weights of large language models (LLMs) from 16-bit to lower bitwidth is the de facto approach to deploy massive transformers onto more affordable accelerators. Whil…