13 citations · 27 across the 10 of their papers we have counts for
8 papers · 1 filter
Probabilistic Inference in Language Models via Twisted Sequential Monte Carlo
Stephen Zhao, Rob Brekelmans, Alireza Makhzani +1
Numerous capability and safety techniques of Large Language Models (LLMs), including RLHF, automated red-teaming, prompt engineering, and infilling, can be cast as sampling from an…
Can We Remove the Square-Root in Adaptive Gradient Methods? A Second-Order Perspective
Wu Lin, Felix Dangel, Runa Eschenhagen +3
Adaptive gradient optimizers like Adam(W) are the default training algorithms for many deep learning architectures, such as transformers. Their diagonal preconditioner is based on…
Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFAC
Wu Lin, Felix Dangel, Runa Eschenhagen +4
Second-order methods such as KFAC can be useful for neural net training. However, they are often memory-inefficient since their preconditioning Kronecker factors are dense, and num…
A Computational Framework for Solving Wasserstein Lagrangian Flows
Kirill Neklyudov, Rob Brekelmans, Alexander Tong +3
The dynamical formulation of the optimal transport can be extended through various choices of the underlying geometry (kinetic energy), and the regularization of density paths (pot…
Improving Mutual Information Estimation with Annealed and Energy-Based Bounds
Rob Brekelmans, Sicong Huang, Marzyeh Ghassemi +3
Mutual information (MI) is a fundamental quantity in information theory and machine learning. However, direct estimation of MI is intractable, even if the true joint probability de…
Action Matching: Learning Stochastic Dynamics from Samples
Kirill Neklyudov, Rob Brekelmans, Daniel Severo +1
Learning the continuous dynamics of a system from snapshots of its temporal marginals is a problem which appears throughout natural sciences and machine learning, including in quan…
Likelihood Ratio Exponential Families
Rob Brekelmans, Frank Nielsen, Alireza Makhzani +2
The exponential family is well known in machine learning and statistical physics as the maximum entropy distribution subject to a set of observed constraints, while the geometric m…
Evaluating Lossy Compression Rates of Deep Generative Models
Sicong Huang, Alireza Makhzani, Yanshuai Cao +1
The field of deep generative modeling has succeeded in producing astonishingly realistic-seeming images and audio, but quantitative evaluation remains a challenge. Log-likelihood i…