19 papers
Variable-Width Transformers
Zhaofeng Wu, Oliver Sieberling, Shawn Tan +3
Scaling model size, specifically depth and width, has driven significant progress in transformer-based language models. However, most architectures maintain a constant width across…
High-Rate Quantized Matrix Multiplication II
Or Ordentlich, Yury Polyanskiy
This is the second part of the work investigating quantized matrix multiplication (MatMul). In part I we considered the case of calibration-free quantization, whereas here we discu…
Measure-to-measure Regression with Transformers
Matthew Vandergrift, Martha White, Yury Polyanskiy +2
Many learning problems require predicting how populations evolve under an unknown transformation. A natural representation for such populations is a probability measure, with point…
Representation Alignment Rests on Linear Structure
Kiril Bangachev, Guy Bresler, Yury Polyanskiy
We investigate the Platonic Representation Hypothesis (PRH) through a tripartite statistical framework of representations: signal, bias, and noise. {1) Signal:} We propose that Pla…
Is Dimensionality a Barrier for Retrieval Models?
Kiril Bangachev, Guy Bresler, Jonathan Kogan +1
Why does the low dimensionality of representations, typically , not prevent modern embedding-based retrieval models from scaling to billions, or even trillions, of d…
Continuous First, Discrete Later: VQ-VAEs Without Dimensional Collapse
Xinyu Zhao, Nikita Karagodin, Hamed Hassani +3
While many approaches to improve VQ-VAE performance focus on codebook size and utilization, the effect of dimensional collapse, where trained VQ-VAE representations live in an extr…
Scaling Limits of Long-Context Transformers
Giuseppe Bruno, Shi Chen, Zhengjiang Lin +2
We study the long-context limit of softmax self-attention with a fixed query and a random context of i.i.d. keys on the sphere, viewing the inverse temperature as the sca…
WaterSIC: Information-Theoretically (Near) Optimal Linear Layer Quantization
Egor Lifar, Semyon Savkin, Or Ordentlich +1
This paper considers the problem of converting a given dense linear layer to low precision. The tradeoff between compressed length and output discrepancy is analyzed information th…
Universal priors: solving empirical Bayes via Bayesian inference and pretraining
Nick Cannella, Anzo Teh, Yanjun Han +1
We theoretically justify the recent empirical finding of [Teh et al., 2025] that a transformer pretrained on synthetically generated data achieves strong performance on empirical B…
Price of metric universality in vector quantization is at most 0.11 bit
Alina Harbuzova, Or Ordentlich, Yury Polyanskiy
Fast computation of a matrix product is a workhorse of modern LLMs. To make their deployment more efficient, a popular approach is that of using a low-precision approxim…
Function estimation in the empirical Bayes setting
Benjamin Kang, Yury Polyanskiy, Anzo Teh
We study function estimation in the empirical Bayes setting for Poisson and normal means. Specifically, given observations with latent parameters $θ_i\sim π…
High-Rate Quantized Matrix Multiplication I
Or Ordentlich, Yury Polyanskiy
This paper investigates the problem of quantized matrix multiplication (MatMul), which has become crucial for the efficient deployment of large language models (LLMs). We consider…
How much can we learn from quantum random circuit sampling?
Tudor Manole, Daniel K. Mark, Wenjie Gong +3
Benchmarking quantum devices is a foundational task for the sustained development of quantum technologies. However, accurate in situ characterization of large-scale quantum devices…
A Gapped Scale-Sensitive Dimension and Lower Bounds for Offset Rademacher Complexity
Zeyu Jia, Yury Polyanskiy, Alexander Rakhlin
We study gapped scale-sensitive dimensions of a function class in both sequential and non-sequential settings. We demonstrate that covering numbers for any uniformly bounded class…
Testing and estimation in orthosymmetric Gaussian sequence model
Zeyu Jia, Yury Polyanskiy
We study the Gaussian sequence model, i.e. , where is assumed to be convex and compact. We show that goodness-of-fit te…
Nonparametric MLE for Gaussian Location Mixtures: Certified Computation and Generic Behavior
Yury Polyanskiy, Mark Sellke
We study the nonparametric maximum likelihood estimator for Gaussian location mixtures in one dimension. It has been known since (Lindsay, 1983) that given an -point…
On the Minimax Regret of Sequential Probability Assignment via Square-Root Entropy
Zeyu Jia, Yury Polyanskiy, Alexander Rakhlin
We study the problem of sequential probability assignment under logarithmic loss, both with and without side information. Our objective is to analyze the minimax regret -- a notion…
Solving Empirical Bayes via Transformers
Anzo Teh, Mark Jabbour, Yury Polyanskiy
This work applies modern AI tools (transformers) to solving one of the oldest statistical problems: Poisson means under empirical Bayes (Poisson-EB) setting. In Poisson-EB a high-d…
NestQuant: Nested Lattice Quantization for Matrix Products and LLMs
Semyon Savkin, Eitan Porat, Or Ordentlich +1
Post-training quantization (PTQ) has emerged as a critical technique for efficient deployment of large language models (LLMs). This work proposes NestQuant, a novel PTQ scheme for…