DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures

Abstract

Recent breakthroughs in Neural Architectural Search (NAS) have achieved state-of-the-art performances in applications such as image classification and language modeling. However, these techniques typically ignore device-related objectives such as inference time, memory usage, and power consumption. Optimizing neural architecture for device-related objectives is immensely crucial for deploying deep networks on portable devices with limited computing resources. We propose DPP-Net: Device-aware Progressive Search for Pareto-optimal Neural Architectures, optimizing for both device-related (e.g., inference time and memory usage) and device-agnostic (e.g., accuracy and model size) objectives. DPP-Net employs a compact search space inspired by current state-of-the-art mobile CNNs, and further improves search efficiency by adopting progressive search (Liu et al. 2017). Experimental results on CIFAR-10 are poised to demonstrate the effectiveness of Pareto-optimal networks found by DPP-Net, for three different devices: (1) a workstation with Titan X GPU, (2) NVIDIA Jetson TX1 embedded system, and (3) mobile phone with ARM Cortex-A53. Compared to CondenseNet and NASNet (Mobile), DPP-Net achieves better performances: higher accuracy and shorter inference time on various devices. Additional experimental results show that models found by DPP-Net also achieve considerably-good performance on ImageNet as well.

arXiv comment13 pages 9 figures, ECCV 2018 Camera Ready

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• Neural Architecture Search with Reinforcement Learning
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Efficient Neural Architecture Search via Parameter Sharing
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• Efficient Architecture Search by Network Transformation
• Progressive Neural Architecture Search
• CondenseNet: An Efficient DenseNet using Learned Group Convolutions

Cited by in corpus (9)

• ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware
• CARS: Continuous Evolution for Efficient Neural Architecture Search
• RC-DARTS: Resource Constrained Differentiable Architecture Search
• Learnable Embedding Space for Efficient Neural Architecture Compression
• Fine-Grained Neural Architecture Search
• Single-Path NAS: Device-Aware Efficient ConvNet Design
• Multi-objective Neural Architecture Search via Non-stationary Policy Gradient
• Distilling with Performance Enhanced Students
• TEA-DNN: the Quest for Time-Energy-Accuracy Co-optimized Deep Neural Networks