An Approximation Algorithm for Optimal Subarchitecture Extraction

Abstract

We consider the problem of finding the set of architectural parameters for a chosen deep neural network which is optimal under three metrics: parameter size, inference speed, and error rate. In this paper we state the problem formally, and present an approximation algorithm that, for a large subset of instances behaves like an FPTAS with an approximation error of $\rho \le |1-\epsilon |$, and that runs in $O(|\Xi |+|W_T^{\ast }|(1+|\Theta ||B||\Xi |/(\epsilon s^{3/2})))$ steps, where $\epsilon$ and $s$ are input parameters; $|B|$ is the batch size; $|W_T^{\ast }|$ denotes the cardinality of the largest weight set assignment; and $|\Xi |$ and $|\Theta |$ are the cardinalities of the candidate architecture and hyperparameter spaces, respectively.

arXiv commentPreprint. Under review. Original submission does not present the bibliography issues from this version

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Neural Architecture Search: A Survey
• On the Convergence of Adam and Beyond
• Validating quantum computers using randomized model circuits
• Making Gradient Descent Optimal for Strongly Convex Stochastic Optimization
• Comparing Rewinding and Fine-tuning in Neural Network Pruning
• Dual Path Networks
• Geometry-Aware Gradient Algorithms for Neural Architecture Search
• On the Bounds of Function Approximations

Cited by in corpus (1)

• Optimal Subarchitecture Extraction For BERT