Generating Neural Networks with Neural Networks
arXiv:1801.01952
Abstract
Hypernetworks are neural networks that generate weights for another neural network. We formulate the hypernetwork training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We explain how this simple formulation generalizes variational inference. We use multi-layered perceptrons to form the mapping from the low dimensional input random vector to the high dimensional weight space, and demonstrate how to reduce the number of parameters in this mapping by parameter sharing. We perform experiments and show that the generated weights are diverse and lie on a non-trivial manifold.
References in corpus (17)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
- Estimating Mutual Information
- Neural Architecture Search with Reinforcement Learning
- Weight Uncertainty in Neural Networks
- Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
- Generating Videos with Scene Dynamics
- The Loss Surfaces of Multilayer Networks
- Predicting Parameters in Deep Learning
- Learning feed-forward one-shot learners
- Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- Essentially No Barriers in Neural Network Energy Landscape
- Learning to Optimize
- ChemGAN challenge for drug discovery: can AI reproduce natural chemical diversity?
- Stochastic Hyperparameter Optimization through Hypernetworks
- Diversified Texture Synthesis with Feed-forward Networks