Efficient Deep Neural Networks
arXiv:1908.08926
Abstract
The success of deep neural networks (DNNs) is attributable to three factors: increased compute capacity, more complex models, and more data. These factors, however, are not always present, especially for edge applications such as autonomous driving, augmented reality, and internet-of-things. Training DNNs requires a large amount of data, which is difficult to obtain. Edge devices such as mobile phones have limited compute capacity, and therefore, require specialized and efficient DNNs. However, due to the enormous design space and prohibitive training costs, designing efficient DNNs for different target devices is challenging. So the question is, with limited data, compute capacity, and model complexity, can we still successfully apply deep neural networks? This dissertation focuses on the above problems and improving the efficiency of deep neural networks at four levels. Model efficiency: we designed neural networks for various computer vision tasks and achieved more than 10x faster speed and lower energy. Data efficiency: we developed an advanced tool that enables 6.2x faster annotation of a LiDAR point cloud. We also leveraged domain adaptation to utilize simulated data, bypassing the need for real data. Hardware efficiency: we co-designed neural networks and hardware accelerators and achieved 11.6x faster inference. Design efficiency: the process of finding the optimal neural networks is time-consuming. Our automated neural architecture search algorithms discovered, using 421x lower computational cost than previous search methods, models with state-of-the-art accuracy and efficiency.
arXiv commentPh.D. dissertation of Bichen Wu
References in corpus (47)
- Adam: A Method for Stochastic Optimization
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
- SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size
- Neural Architecture Search with Reinforcement Learning
- Categorical Reparameterization with Gumbel-Softmax
- Instance Normalization: The Missing Ingredient for Fast Stylization
- Efficient Inference in Fully Connected CRFs with Gaussian Edge Potentials
- Learning Transferable Features with Deep Adaptation Networks
- Conditional Random Fields as Recurrent Neural Networks
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
- Going Deeper with Convolutions
- DARTS: Differentiable Architecture Search
- AMC: AutoML for Model Compression and Acceleration on Mobile Devices
- The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables
- cuDNN: Efficient Primitives for Deep Learning
- Trained Ternary Quantization
- PACT: Parameterized Clipping Activation for Quantized Neural Networks
- DenseNet: Implementing Efficient ConvNet Descriptor Pyramids
- Efficient Neural Architecture Search via Parameter Sharing
- Achieving Human Parity on Automatic Chinese to English News Translation
- Curriculum Domain Adaptation for Semantic Segmentation of Urban Scenes
- Aggregated Residual Transformations for Deep Neural Networks
- Xception: Deep Learning with Depthwise Separable Convolutions
- Domain Adaptation for Visual Applications: A Comprehensive Survey
- Vehicle Detection from 3D Lidar Using Fully Convolutional Network
- ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design
- Single Path One-Shot Neural Architecture Search with Uniform Sampling
- Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search
- DetNAS: Backbone Search for Object Detection
- TSM: Temporal Shift Module for Efficient Video Understanding
- Progressive Neural Architecture Search
- Label Refinery: Improving ImageNet Classification through Label Progression
- PointSeg: Real-Time Semantic Segmentation Based on 3D LiDAR Point Cloud
- Tiny-DSOD: Lightweight Object Detection for Resource-Restricted Usages
- SqueezeDet: Unified, Small, Low Power Fully Convolutional Neural Networks for Real-Time Object Detection for Autonomous Driving
- Visual Wake Words Dataset
- Multi-View 3D Object Detection Network for Autonomous Driving
- Shift: A Zero FLOP, Zero Parameter Alternative to Spatial Convolutions
- SqueezeSegV2: Improved Model Structure and Unsupervised Domain Adaptation for Road-Object Segmentation from a LiDAR Point Cloud
- HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision
- Towards Effective Low-bitwidth Convolutional Neural Networks
- Subcategory-aware Convolutional Neural Networks for Object Proposals and Detection
- Training Compact Neural Networks with Binary Weights and Low Precision Activations
- LATTE: Accelerating LiDAR Point Cloud Annotation via Sensor Fusion, One-Click Annotation, and Tracking