Principled Weight Initialization for Hypernetworks

Abstract

Hypernetworks are meta neural networks that generate weights for a main neural network in an end-to-end differentiable manner. Despite extensive applications ranging from multi-task learning to Bayesian deep learning, the problem of optimizing hypernetworks has not been studied to date. We observe that classical weight initialization methods like Glorot & Bengio (2010) and He et al. (2015), when applied directly on a hypernet, fail to produce weights for the mainnet in the correct scale. We develop principled techniques for weight initialization in hypernets, and show that they lead to more stable mainnet weights, lower training loss, and faster convergence.

References in corpus (15)

• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• On the Convergence of Adam and Beyond
• Exact solutions to the nonlinear dynamics of learning in deep linear neural networks
• Hypernetwork Knowledge Graph Embeddings
• Graph HyperNetworks for Neural Architecture Search
• Fixup Initialization: Residual Learning Without Normalization
• MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning
• Stochastic Hyperparameter Optimization through Hypernetworks
• Hypernetwork functional image representation
• Implicit Weight Uncertainty in Neural Networks
• Meta Networks for Neural Style Transfer
• HyperST-Net: Hypernetworks for Spatio-Temporal Forecasting
• HyperNetworks with statistical filtering for defending adversarial examples
• Predictive Uncertainty Quantification with Compound Density Networks
• A Generative Model for Sampling High-Performance and Diverse Weights for Neural Networks

Cited by in corpus (17)

• Learning the Effect of Registration Hyperparameters with HyperMorph
• HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling
• Controllable Pareto Multi-Task Learning
• On the Modularity of Hypernetworks
• Infinitely Deep Bayesian Neural Networks with Stochastic Differential Equations
• DHP: Differentiable Meta Pruning via HyperNetworks
• Regularization-Agnostic Compressed Sensing MRI Reconstruction with Hypernetworks
• On Infinite-Width Hypernetworks
• CoNeS: Conditional neural fields with shift modulation for multi-sequence MRI translation
• Posterior Meta-Replay for Continual Learning
• Parameter Prediction for Unseen Deep Architectures
• Differential Evolution Algorithm based Hyper-Parameters Selection of Convolutional Neural Network for Speech Command Recognition
• Block Contextual MDPs for Continual Learning
• The Heterogeneity Hypothesis: Finding Layer-Wise Differentiated Network Architectures
• HyperHyperNetworks for the Design of Antenna Arrays
• Recomposing the Reinforcement Learning Building Blocks with Hypernetworks
• WeightAlign: Normalizing Activations by Weight Alignment