145 citations · 448 across the 24 of their papers we have counts for
24 papers
Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data
Shikai Qiu, Marc Finzi, Yujia Zheng +2
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networ…
From Entropy to Epiplexity: Rethinking Information for Computationally Bounded Intelligence
Marc Finzi, Shikai Qiu, Yiding Jiang +3
Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to exist…
Compute-Optimal LLMs Provably Generalize Better With Scale
Marc Finzi, Sanyam Kapoor, Diego Granziol +4
Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the…
Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices
Andres Potapczynski, Shikai Qiu, Marc Finzi +6
Dense linear layers are the dominant computational bottleneck in large neural networks, presenting a critical need for more efficient alternatives. Previous efforts focused on a sm…
Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Models
Sanae Lotfi, Yilun Kuang, Brandon Amos +3
Large language models (LLMs) with billions of parameters excel at predicting the next token in a sequence. Recent work computes non-vacuous compression-based generalization bounds…
Compute Better Spent: Replacing Dense Layers with Structured Matrices
Shikai Qiu, Andres Potapczynski, Marc Finzi +2
Dense linear layers are the dominant computational bottleneck in foundation models. Identifying more efficient alternatives to dense matrices has enormous potential for building mo…
Non-Vacuous Generalization Bounds for Large Language Models
Sanae Lotfi, Marc Finzi, Yilun Kuang +3
Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We…
Large Language Models Are Zero-Shot Time Series Forecasters
Nate Gruver, Marc Finzi, Shikai Qiu +1
By encoding time series as a string of numerical digits, we can frame time series forecasting as next-token prediction in text. Developing this approach, we find that large languag…
CoLA: Exploiting Compositional Structure for Automatic and Efficient Numerical Linear Algebra
Andres Potapczynski, Marc Finzi, Geoff Pleiss +1
Many areas of machine learning and science involve large linear algebra problems, such as eigendecompositions, solving linear systems, computing matrix exponentials, and trace esti…
User-defined Event Sampling and Uncertainty Quantification in Diffusion Models for Physical Dynamical Systems
Marc Finzi, Anudhyan Boral, Andrew Gordon Wilson +2
Diffusion models are a class of probabilistic generative models that have been widely used as a prior for image processing tasks like text conditional generation and inpainting. We…
A Stable and Scalable Method for Solving Initial Value PDEs with Neural Networks
Marc Finzi, Andres Potapczynski, Matthew Choptuik +1
Unlike conventional grid and mesh based methods for solving partial differential equations (PDEs), neural networks have the potential to break the curse of dimensionality, providin…
The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learning
Micah Goldblum, Marc Finzi, Keefer Rowan +1
No free lunch theorems for supervised learning state that no learner can solve all problems or that all learners achieve exactly the same accuracy on average over a uniform distrib…
PAC-Bayes Compression Bounds So Tight That They Can Explain Generalization
Sanae Lotfi, Marc Finzi, Sanyam Kapoor +3
While there has been progress in developing non-vacuous generalization bounds for deep neural networks, these bounds tend to be uninformative about why deep learning works. In this…
The Lie Derivative for Measuring Learned Equivariance
Nate Gruver, Marc Finzi, Micah Goldblum +1
Equivariance guarantees that a model's predictions capture key symmetries in data. When an image is translated or rotated, an equivariant model's representation of that image will…
Deconstructing the Inductive Biases of Hamiltonian Neural Networks
Nate Gruver, Marc Finzi, Samuel Stanton +1
Physics-inspired neural networks (NNs), such as Hamiltonian or Lagrangian NNs, dramatically outperform other learned dynamics models by leveraging strong inductive biases. These mo…
Residual Pathway Priors for Soft Equivariance Constraints
Marc Finzi, Gregory Benton, Andrew Gordon Wilson
There is often a trade-off between building deep learning systems that are expressive enough to capture the nuances of the reality, and having the right inductive biases for effici…
SKIing on Simplices: Kernel Interpolation on the Permutohedral Lattice for Scalable Gaussian Processes
Sanyam Kapoor, Marc Finzi, Ke Alexander Wang +1
State-of-the-art methods for scalable Gaussian processes use iterative algorithms, requiring fast matrix vector multiplies (MVMs) with the covariance kernel. The Structured Kernel…
A Practical Method for Constructing Equivariant Multilayer Perceptrons for Arbitrary Matrix Groups
Marc Finzi, Max Welling, Andrew Gordon Wilson
Symmetries and equivariance are fundamental to the generalization of neural networks on domains such as images, graphs, and point clouds. Existing work has primarily focused on a s…
Learning Invariances in Neural Networks
Gregory Benton, Marc Finzi, Pavel Izmailov +1
Invariances to translations have imbued convolutional neural networks with powerful generalization properties. However, we often do not know a priori what invariances are present i…
Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints
Marc Finzi, Ke Alexander Wang, Andrew Gordon Wilson
Reasoning about the physical world requires models that are endowed with the right inductive biases to learn the underlying dynamics. Recent works improve generalization for predic…
Probabilistic Numeric Convolutional Neural Networks
Marc Finzi, Roberto Bondesan, Max Welling
Continuous input signals like images and time series that are irregularly sampled or have missing values are challenging for existing deep learning methods. Coherently defined feat…
Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data
Marc Finzi, Samuel Stanton, Pavel Izmailov +1
The translation equivariance of convolutional layers enables convolutional neural networks to generalize well on image problems. While translation equivariance provides a powerful…
Semi-Supervised Learning with Normalizing Flows
Pavel Izmailov, Polina Kirichenko, Marc Finzi +1
Normalizing flows transform a latent distribution through an invertible neural network for a flexible and pleasingly simple approach to generative modelling, while preserving an ex…
There Are Many Consistent Explanations of Unlabeled Data: Why You Should Average
Ben Athiwaratkun, Marc Finzi, Pavel Izmailov +1
Presently the most successful approaches to semi-supervised learning are based on consistency regularization, whereby a model is trained to be robust to small perturbations of its…