Towards NNGP-guided Neural Architecture Search

Abstract

The predictions of wide Bayesian neural networks are described by a Gaussian process, known as the Neural Network Gaussian Process (NNGP). Analytic forms for NNGP kernels are known for many models, but computing the exact kernel for convolutional architectures is prohibitively expensive. One can obtain effective approximations of these kernels through Monte-Carlo estimation using finite networks at initialization. Monte-Carlo NNGP inference is orders-of-magnitude cheaper in FLOPs compared to gradient descent training when the dataset size is small. Since NNGP inference provides a cheap measure of performance of a network architecture, we investigate its potential as a signal for neural architecture search (NAS). We compute the NNGP performance of approximately 423k networks in the NAS-bench 101 dataset on CIFAR-10 and compare its utility against conventional performance measures obtained by shortened gradient-based training. We carry out a similar analysis on 10k randomly sampled networks in the mobile neural architecture search (MNAS) space for ImageNet. We discover comparative advantages of NNGP-based metrics, and discuss potential applications. In particular, we propose that NNGP performance is an inexpensive signal independent of metrics obtained from training that can either be used for reducing big search spaces, or improving training-based performance measures.

arXiv comment13 + 6 pages, 19 figures; open-source code available at https://github.com/google-research/google-research/tree/master/nngp_nas

References in corpus (26)

• Neural Architecture Search with Reinforcement Learning
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• A Convergence Theory for Deep Learning via Over-Parameterization
• Efficient Neural Architecture Search via Parameter Sharing
• Gradient Descent Provably Optimizes Over-parameterized Neural Networks
• Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Data
• NAS-Bench-101: Towards Reproducible Neural Architecture Search
• Toward Deeper Understanding of Neural Networks: The Power of Initialization and a Dual View on Expressivity
• Scaling Limits of Wide Neural Networks with Weight Sharing: Gaussian Process Behavior, Gradient Independence, and Neural Tangent Kernel Derivation
• Bayesian Deep Convolutional Networks with Many Channels are Gaussian Processes
• Enhanced Convolutional Neural Tangent Kernels
• SGD Learns the Conjugate Kernel Class of the Network
• Neural Tangents: Fast and Easy Infinite Neural Networks in Python
• Steps Toward Deep Kernel Methods from Infinite Neural Networks
• Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach
• Accelerating Neural Architecture Search using Performance Prediction
• Neural Architecture Search without Training
• On the Selection of Initialization and Activation Function for Deep Neural Networks
• Provable Benefit of Orthogonal Initialization in Optimizing Deep Linear Networks
• Neural Kernels Without Tangents
• On the Neural Tangent Kernel of Deep Networks with Orthogonal Initialization
• On the infinite width limit of neural networks with a standard parameterization
• A Gaussian Process perspective on Convolutional Neural Networks
• On the expected behaviour of noise regularised deep neural networks as Gaussian processes
• Critical initialisation for deep signal propagation in noisy rectifier neural networks
• A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

Cited by in corpus (7)

• Neural Architecture Search without Training
• Evaluating Efficient Performance Estimators of Neural Architectures
• Dataset Distillation with Infinitely Wide Convolutional Networks
• Generalization bounds for deep learning
• GradSign: Model Performance Inference with Theoretical Insights
• Generalization Guarantees for Neural Architecture Search with Train-Validation Split
• Contrastive Embeddings for Neural Architectures