Parameter Prediction for Unseen Deep Architectures

Abstract

Deep learning has been successful in automating the design of features in machine learning pipelines. However, the algorithms optimizing neural network parameters remain largely hand-designed and computationally inefficient. We study if we can use deep learning to directly predict these parameters by exploiting the past knowledge of training other networks. We introduce a large-scale dataset of diverse computational graphs of neural architectures - DeepNets-1M - and use it to explore parameter prediction on CIFAR-10 and ImageNet. By leveraging advances in graph neural networks, we propose a hypernetwork that can predict performant parameters in a single forward pass taking a fraction of a second, even on a CPU. The proposed model achieves surprisingly good performance on unseen and diverse networks. For example, it is able to predict all 24 million parameters of a ResNet-50 achieving a 60% accuracy on CIFAR-10. On ImageNet, top-5 accuracy of some of our networks approaches 50%. Our task along with the model and results can potentially lead to a new, more computationally efficient paradigm of training networks. Our model also learns a strong representation of neural architectures enabling their analysis.

NeurIPS 2021 camera ready, the code is available at https://github.com/facebookresearch/ppuda

References in corpus (50)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• An overview of gradient descent optimization algorithms
• The architecture of complex weighted networks
• Neural Architecture Search with Reinforcement Learning
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• Language Models are Few-Shot Learners
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Wide Residual Networks
• AutoML: A Survey of the State-of-the-Art
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Transfusion: Understanding Transfer Learning for Medical Imaging
• Gated Graph Sequence Neural Networks
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• A Survey on Graph Kernels
• What makes ImageNet good for transfer learning?
• Rigid-Motion Scattering for Texture Classification
• High-Performance Large-Scale Image Recognition Without Normalization
• Learning feed-forward one-shot learners
• Benchmarking Graph Neural Networks
• What is being transferred in transfer learning?
• A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
• DAG-GNN: DAG Structure Learning with Graph Neural Networks
• Graph HyperNetworks for Neural Architecture Search
• Fixup Initialization: Residual Learning Without Normalization
• Learned Optimizers that Scale and Generalize
• On the Bottleneck of Graph Neural Networks and its Practical Implications
• Position-aware Graph Neural Networks
• FreezeOut: Accelerate Training by Progressively Freezing Layers
• Batch Normalization is a Cause of Adversarial Vulnerability
• LGM-Net: Learning to Generate Matching Networks for Few-Shot Learning
• BigNAS: Scaling Up Neural Architecture Search with Big Single-Stage Models
• Principled Weight Initialization for Hypernetworks
• Faster Neural Network Training with Data Echoing
• Non-Local Graph Neural Networks
• HyperGAN: A Generative Model for Diverse, Performant Neural Networks
• Characterizing signal propagation to close the performance gap in unnormalized ResNets
• Progressive DARTS: Bridging the Optimization Gap for NAS in the Wild
• Neural Predictor for Neural Architecture Search
• Tasks, stability, architecture, and compute: Training more effective learned optimizers, and using them to train themselves
• SPAGAN: Shortest Path Graph Attention Network
• GradInit: Learning to Initialize Neural Networks for Stable and Efficient Training
• Meta Learning Backpropagation And Improving It
• Building LEGO Using Deep Generative Models of Graphs
• Meta Learning Black-Box Population-Based Optimizers
• Are wider nets better given the same number of parameters?
• Data-driven Weight Initialization with Sylvester Solvers