HyperNetworks

Abstract

This work explores hypernetworks: an approach of using a one network, also known as a hypernetwork, to generate the weights for another network. Hypernetworks provide an abstraction that is similar to what is found in nature: the relationship between a genotype - the hypernetwork - and a phenotype - the main network. Though they are also reminiscent of HyperNEAT in evolution, our hypernetworks are trained end-to-end with backpropagation and thus are usually faster. The focus of this work is to make hypernetworks useful for deep convolutional networks and long recurrent networks, where hypernetworks can be viewed as relaxed form of weight-sharing across layers. Our main result is that hypernetworks can generate non-shared weights for LSTM and achieve near state-of-the-art results on a variety of sequence modelling tasks including character-level language modelling, handwriting generation and neural machine translation, challenging the weight-sharing paradigm for recurrent networks. Our results also show that hypernetworks applied to convolutional networks still achieve respectable results for image recognition tasks compared to state-of-the-art baseline models while requiring fewer learnable parameters.

Cited by in corpus (15)

• A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges
• Hypernetwork functional image representation
• Dynamic Instance Domain Adaptation
• HAMUR: Hyper Adapter for Multi-Domain Recommendation
• Learning the Effect of Registration Hyperparameters with HyperMorph
• Towards constraining warm dark matter with stellar streams through neural simulation-based inference
• Transforming task representations to perform novel tasks
• Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules
• PriMask: Cascadable and Collusion-Resilient Data Masking for Mobile Cloud Inference
• Data Consistent Deep Rigid MRI Motion Correction
• Modularity in Deep Learning: A Survey
• HOI-aware Adaptive Network for Weakly-supervised Action Segmentation
• HyperNCA: Growing Developmental Networks with Neural Cellular Automata
• ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution
• WeightNet: Revisiting the Design Space of Weight Networks