Peephole: Predicting Network Performance Before Training

Abstract

The quest for performant networks has been a significant force that drives the advancements of deep learning in recent years. While rewarding, improving network design has never been an easy journey. The large design space combined with the tremendous cost required for network training poses a major obstacle to this endeavor. In this work, we propose a new approach to this problem, namely, predicting the performance of a network before training, based on its architecture. Specifically, we develop a unified way to encode individual layers into vectors and bring them together to form an integrated description via LSTM. Taking advantage of the recurrent network's strong expressive power, this method can reliably predict the performances of various network architectures. Our empirical studies showed that it not only achieved accurate predictions but also produced consistent rankings across datasets -- a key desideratum in performance prediction.

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Residual Learning for Image Recognition
• Neural Architecture Search with Reinforcement Learning
• Densely Connected Convolutional Networks
• Going Deeper with Convolutions
• Visualizing and Understanding Recurrent Networks
• Aggregated Residual Transformations for Deep Neural Networks
• Freeze-Thaw Bayesian Optimization
• Practical Block-wise Neural Network Architecture Generation

Cited by in corpus (26)

• AutoML: A Survey of the State-of-the-Art
• Efficient Neural Architecture Search via Parameter Sharing
• SNAS: Stochastic Neural Architecture Search
• A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
• Graph HyperNetworks for Neural Architecture Search
• Neural Predictor based Quantum Architecture Search
• A Comprehensive Survey on Hardware-Aware Neural Architecture Search
• Neural Architecture Search without Training
• A Learned Performance Model for Tensor Processing Units
• Evaluating Efficient Performance Estimators of Neural Architectures
• Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap
• Perceptron Theory Can Predict the Accuracy of Neural Networks
• ReNAS:Relativistic Evaluation of Neural Architecture Search
• A Semi-Supervised Assessor of Neural Architectures
• Feature Partitioning for Efficient Multi-Task Architectures
• Fast Task-Aware Architecture Inference
• EcoNAS: Finding Proxies for Economical Neural Architecture Search
• TAPAS: Train-less Accuracy Predictor for Architecture Search
• NeuNetS: An Automated Synthesis Engine for Neural Network Design
• DC-NAS: Divide-and-Conquer Neural Architecture Search
• Trainless Model Performance Estimation for Neural Architecture Search
• A Novel Training Protocol for Performance Predictors of Evolutionary Neural Architecture Search Algorithms
• Data Proxy Generation for Fast and Efficient Neural Architecture Search
• Cascade Bagging for Accuracy Prediction with Few Training Samples
• "You might also like this model": Data Driven Approach for Recommending Deep Learning Models for Unknown Image Datasets
• GPNAS: A Neural Network Architecture Search Framework Based on Graphical Predictor