A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks
arXiv:1905.02898
Abstract
Recent work on mode connectivity in the loss landscape of deep neural networks has demonstrated that the locus of (sub-)optimal weight vectors lies on continuous paths. In this work, we train a neural network that serves as a hypernetwork, mapping a latent vector into high-performance (low-loss) weight vectors, generalizing recent findings of mode connectivity to higher dimensional manifolds. We formulate the training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We demonstrate how to reduce the number of parameters in the hypernetwork by parameter sharing. Once learned, the hypernetwork allows for a computationally efficient, ancestral sampling of neural network weights, which we recruit to form large ensembles. The improvement in classification accuracy obtained by this ensembling indicates that the generated manifold extends in dimensions other than directions implied by trivial symmetries. For computational efficiency, we distill an ensemble into a single classifier while retaining generalization.
arXiv admin note: substantial text overlap with arXiv:1801.01952
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Estimating Mutual Information
- Neural Architecture Search with Reinforcement Learning
- Weight Uncertainty in Neural Networks
- The Loss Surfaces of Multilayer Networks
- Predicting Parameters in Deep Learning
- Identifying and attacking the saddle point problem in high-dimensional non-convex optimization
- Learning feed-forward one-shot learners
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Essentially No Barriers in Neural Network Energy Landscape
- Learning to Optimize
- Stochastic Hyperparameter Optimization through Hypernetworks
- Diversified Texture Synthesis with Feed-forward Networks