354 citations · 1.9k across the 38 of their papers we have counts for
20 papers · 1 filter
Variational Prediction
Alexander A. Alemi, Ben Poole
Bayesian inference offers benefits over maximum likelihood, but it also comes with computational costs. Computing the posterior is typically intractable, as is marginalizing that p…
Weighted Ensemble Self-Supervised Learning
Yangjun Ruan, Saurabh Singh, Warren Morningstar +4
Ensembling has proven to be a powerful technique for boosting model performance, uncertainty estimation, and robustness in supervised learning. Advances in self-supervised learning…
A Closer Look at the Adversarial Robustness of Information Bottleneck Models
Iryna Korshunova, David Stutz, Alexander A. Alemi +2
We study the adversarial robustness of information bottleneck models for classification. Previous works showed that the robustness of models trained with information bottlenecks ca…
Does Knowledge Distillation Really Work?
Samuel Stanton, Pavel Izmailov, Polina Kirichenko +2
Knowledge distillation is a popular technique for training a small student network to emulate a larger teacher model, such as an ensemble of networks. We show that while knowledge…
PAC-Bayes: Narrowing the Empirical Risk Gap in the Misspecified Bayesian Regime
Warren R. Morningstar, Alexander A. Alemi, Joshua V. Dillon
The Bayesian posterior minimizes the "inferential risk" which itself bounds the "predictive risk". This bound is tight when the likelihood and prior are well-specified. However sin…
Density of States Estimation for Out-of-Distribution Detection
Warren R. Morningstar, Cusuh Ham, Andrew G. Gallagher +3
Perhaps surprisingly, recent studies have shown probabilistic model likelihoods have poor specificity for out-of-distribution (OOD) detection and often assign higher likelihoods to…
CEB Improves Model Robustness
Ian Fischer, Alexander A. Alemi
We demonstrate that the Conditional Entropy Bottleneck (CEB) can improve model robustness. CEB is an easy strategy to implement and works in tandem with data augmentation procedure…
Information in Infinite Ensembles of Infinitely-Wide Neural Networks
Ravid Shwartz-Ziv, Alexander A. Alemi
In this preliminary work, we study the generalization properties of infinite ensembles of infinitely-wide neural networks. Amazingly, this model family admits tractable calculation…
Variational Predictive Information Bottleneck
Alexander A. Alemi
In classic papers, Zellner demonstrated that Bayesian inference could be derived as the solution to an information theoretic functional. Below we derive a generalized form of this…
On Predictive Information in RNNs
Zhe Dong, Deniz Oktay, Ben Poole +1
Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work,…
Dueling Decoders: Regularizing Variational Autoencoder Latent Spaces
Bryan Seybold, Emily Fertig, Alex Alemi +1
Variational autoencoders learn unsupervised data representations, but these models frequently converge to minima that fail to preserve meaningful semantic information. For example,…
On Variational Bounds of Mutual Information
Ben Poole, Sherjil Ozair, Aaron van den Oord +2
Estimating and optimizing Mutual Information (MI) is core to many problems in machine learning; however, bounding MI in high dimensions is challenging. To establish tractable and s…
-VAEs can retain label information even at high compression
Emily Fertig, Aryan Arbabi, Alexander A. Alemi
In this paper, we investigate the degree to which the encoding of a -VAE captures label information across multiple architectures on Binary Static MNIST and Omniglot. Even thoug…
Uncertainty in the Variational Information Bottleneck
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon
We present a simple case study, demonstrating that Variational Information Bottleneck (VIB) can improve a network's classification calibration as well as its ability to detect out-…
TherML: Thermodynamics of Machine Learning
Alexander A. Alemi, Ian Fischer
In this work we offer a framework for reasoning about a wide class of existing objectives in machine learning. We develop a formal correspondence between this work and thermodynami…
TensorFlow Distributions
Joshua V. Dillon, Ian Langmore, Dustin Tran +7
The TensorFlow Distributions library implements a vision of probability theory adapted to the modern deep-learning paradigm of end-to-end differentiable computation. Building on tw…
Fixing a Broken ELBO
Alexander A. Alemi, Ben Poole, Ian Fischer +3
Recent work in unsupervised representation learning has focused on learning deep directed latent-variable models. Fitting these models by maximizing the marginal likelihood or evid…
Watch Your Step: Learning Node Embeddings via Graph Attention
Sami Abu-El-Haija, Bryan Perozzi, Rami Al-Rfou +1
Graph embedding methods represent nodes in a continuous vector space, preserving information from the graph (e.g. by sampling random walks). There are many hyper-parameters to thes…
Improved generator objectives for GANs
Ben Poole, Alexander A. Alemi, Jascha Sohl-Dickstein +1
We present a framework to understand GAN training as alternating density ratio estimation and approximate divergence minimization. This provides an interpretation for the mismatche…
Deep Variational Information Bottleneck
Alexander A. Alemi, Ian Fischer, Joshua V. Dillon +1
We present a variational approximation to the information bottleneck of Tishby et al. (1999). This variational approach allows us to parameterize the information bottleneck model u…