354 citations · 1.9k across the 38 of their papers we have counts for
39 papers
Speed Limits for Deep Learning
Inbar Seroussi, Alexander A. Alemi, Moritz Helias +1
State-of-the-art neural networks require extreme computational power to train. It is therefore natural to wonder whether they are optimally trained. Here we apply a recent advancem…
Variational Prediction
Alexander A. Alemi, Ben Poole
Bayesian inference offers benefits over maximum likelihood, but it also comes with computational costs. Computing the posterior is typically intractable, as is marginalizing that p…
Weighted Ensemble Self-Supervised Learning
Yangjun Ruan, Saurabh Singh, Warren Morningstar +4
Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning…
Bayesian Imitation Learning for End-to-End Mobile Manipulation
Yuqing Du, Daniel Ho, Alexander A. Alemi +2
In this work we investigate and demonstrate benefits of a Bayesian approach to imitation learning from multiple sensor inputs, as applied to the task of opening office doors with a…
A Closer Look at the Adversarial Robustness of Information Bottleneck Models
Iryna Korshunova, David Stutz, Alexander A. Alemi +2
We study the adversarial robustness of information bottleneck models for classification. Previous works showed that the robustness of models trained with information bottlenecks ca…
Does Knowledge Distillation Really Work?
Samuel Stanton, Pavel Izmailov, Polina Kirichenko +2
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge…
VIB is Half Bayes
Alexander A Alemi, Warren R Morningstar, Ben Poole +2
In discriminative settings such as regression and classification there are two random variables at play, the inputs X and the targets Y. Here, we demonstrate that the Variational I…
PAC-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
Warren R. Morningstar, Alexander A. Alemi, Joshua V. Dillon
The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However sin…
Density of States Estimation for Out-of-Distribution Detection
Warren R. Morningstar, Cusuh Ham, Andrew G. Gallagher +3
Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to…
The OpenKIM Processing Pipeline: A Cloud-Based Automatic Materials Property Computation Engine
Daniel S. Karls, Matthew Bierbaum, Alexander A. Alemi +3
The Open Knowledgebase of Interatomic Models (OpenKIM) project is a framework intended to facilitate access to standardized implementations of interatomic models for molecular simu…
CEB Improves Model Robustness
Ian Fischer, Alexander A. Alemi
We demonstrate that the Conditional Entropy Bottleneck (CEB) can improve model robustness. CEB is an easy strategy to implement and works in tandem with data augmentation procedure…
Neural Tangents: Fast and Easy Infinite Neural Networks in Python
Roman Novak, Lechao Xiao, Jiri Hron +4
Neural Tangents is a library designed to enable research into infinite-width neural networks. It provides a high-level API for specifying complex and hierarchical neural network ar…
Thermodynamic Computing
Tom Conte, Erik DeBenedictis, Natesh Ganesh +36
The hardware and software foundations laid in the first half of the 20th Century enabled the computing technologies that have transformed the world, but these foundations are now u…
Information in Infinite Ensembles of Infinitely-Wide Neural Networks
Ravid Shwartz-Ziv, Alexander A. Alemi
In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculation…
Variational Predictive Information Bottleneck
Alexander A. Alemi
In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this…
On Predictive Information in RNNs
Zhe Dong, Deniz Oktay, Ben Poole +1
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work,…
Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
Bryan Seybold, Emily Fertig, Alex Alemi +1
Variational autoencoders learn unsupervised data representations, but these models frequently converge to minima that fail to preserve meaningful semantic information. For example,…
On Variational Bounds of Mutual Information
Ben Poole, Sherjil Ozair, Aaron van den Oord +2
Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and s…
On the Use of ArXiv as a Dataset
Colin B. Clement, Matthew Bierbaum, Kevin P. O'Keeffe +1
The arXiv has collected 1.5 million pre-print articles over 28 years, hosting literature from scientific fields including Physics, Mathematics, and Computer Science. Each pre-print…
-VAEs can retain label information even at high compression
Emily Fertig, Aryan Arbabi, Alexander A. Alemi
In this paper, we investigate the degree to which the encoding of a -VAE captures label information across multiple architectures on Binary Static MNIST and Omniglot. Even thoug…
Uncertainty in the Variational Information Bottleneck
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon
We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-…
TherML: Thermodynamics of Machine Learning
Alexander A. Alemi, Ian Fischer
In this work we offer a framework for reasoning about a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynami…
GILBO: One Metric to Measure Them All
Alexander A. Alemi, Ian Fischer
We propose a simple, tractable lower bound on the mutual information contained in the joint generative density of any latent variable generative model: the GILBO (Generative Inform…
TensorFlow Distributions
Joshua V. Dillon, Ian Langmore, Dustin Tran +7
The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on tw…