HardCoRe-NAS: Hard Constrained diffeRentiable Neural Architecture Search

Abstract

Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. A popular approach to find fitting networks is through constrained Neural Architecture Search (NAS), however, previous methods enforce the constraint only softly. Therefore, the resulting networks do not exactly adhere to the resource constraint and their accuracy is harmed. In this work we resolve this by introducing Hard Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate formulation of the expected resource requirement and a scalable search method that satisfies the hard constraint throughout the search. Our experiments show that HardCoRe-NAS generates state-of-the-art architectures, surpassing other NAS methods, while strictly satisfying the hard resource constraints without any tuning required.

Niv Nayman and Yonathan Aflalo contributed equally. An implementation of HardCoRe-NAS is available at: https://github.com/Alibaba-MIIL/HardCoReNAS

References in corpus (20)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• mixup: Beyond Empirical Risk Minimization
• Neural Architecture Search with Reinforcement Learning
• Categorical Reparameterization with Gumbel-Softmax
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• AutoAugment: Learning Augmentation Policies from Data
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• SNAS: Stochastic Neural Architecture Search
• DARTS+: Improved Differentiable Architecture Search with Early Stopping
• On the Global Linear Convergence of Frank-Wolfe Optimization Variants
• Variance-Reduced and Projection-Free Stochastic Optimization
• How to Train Your Super-Net: An Analysis of Training Heuristics in Weight-Sharing NAS
• Deep Frank-Wolfe For Neural Network Optimization
• Knapsack Pruning with Inner Distillation
• TF-NAS: Rethinking Three Search Freedoms of Latency-Constrained Differentiable Neural Architecture Search
• Projection-Free Adaptive Gradients for Large-Scale Optimization
• Deep Neural Network Training with Frank-Wolfe
• Global Convergence of Frank Wolfe on One Hidden Layer Networks

Cited by in corpus (2)

• Enhancing Neural Architecture Search with Multiple Hardware Constraints for Deep Learning Model Deployment on Tiny IoT Devices
• Layer Folding: Neural Network Depth Reduction using Activation Linearization