HyperGAN: A Generative Model for Diverse, Performant Neural Networks

Abstract

Standard neural networks are often overconfident when presented with data outside the training distribution. We introduce HyperGAN, a new generative model for learning a distribution of neural network parameters. HyperGAN does not require restrictive assumptions on priors, and networks sampled from it can be used to quickly create very large and diverse ensembles. HyperGAN employs a novel mixer to project prior samples to a latent space with correlated dimensions, and samples from the latent space are then used to generate weights for each layer of a deep neural network. We show that HyperGAN can learn to generate parameters which label the MNIST and CIFAR-10 datasets with competitive performance to fully supervised learning, while learning a rich distribution of effective parameters. We also show that HyperGAN can also provide better uncertainty estimates than standard ensembles by evaluating on out of distribution data as well as adversarial examples.

arXiv comment11 pages, 10 figures, 6 tables, Accepted to ICML 2019

References in corpus (7)

• Popular Ensemble Methods: An Empirical Study
• The Loss Surfaces of Multilayer Networks
• Probabilistic Backpropagation for Scalable Learning of Bayesian Neural Networks
• Using Fast Weights to Attend to the Recent Past
• Stochastic Hyperparameter Optimization through Hypernetworks
• Implicit Weight Uncertainty in Neural Networks
• Adversarial Distillation of Bayesian Neural Network Posteriors

Cited by in corpus (7)

• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Learning the Effect of Registration Hyperparameters with HyperMorph
• PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inference
• Graceful Degradation and Related Fields
• Implicit Generative Modeling for Efficient Exploration
• Parameter Prediction for Unseen Deep Architectures
• Meta Internal Learning