19 papers
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Nick Cannella, Anzo Teh, Yanjun Han +1
We theoretically justify the recent empirical finding of [Teh et al., 2025] that a transformer pretrained on synthetically generated data achieves strong performance on empirical B…
Variable-Width Transformers
Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…
Price of metric universality in vector quantization is at most 0.11 bit
Alina Harbuzova, Or Ordentlich, Yury Polyanskiy
Fast computation of a matrix product is a workhorse of modern LLMs. To make their deployment more efficient, a popular approach is that of using a low-precision approxim…
High-Rate Quantized Matrix Multiplication II
Or Ordentlich, Yury Polyanskiy
This is the second part of the work investigating quantized matrix multiplication (MatMul). In part I we considered the case of calibration-free quantization, whereas here we discu…
WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization
Egor Lifar, Semyon Savkin, Or Ordentlich +1
This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information th…
Measure-to-measure Regression with Transformers
Matthew Vandergrift, Martha White, Yury Polyanskiy +2
Many learning problems require predicting how populations evolve under an unknown transformation. A natural representation for such populations is a probability measure, with point…