GeneCAI: Genetic Evolution for Acquiring Compact AI

Abstract

In the contemporary big data realm, Deep Neural Networks (DNNs) are evolving towards more complex architectures to achieve higher inference accuracy. Model compression techniques can be leveraged to efficiently deploy such compute-intensive architectures on resource-limited mobile devices. Such methods comprise various hyper-parameters that require per-layer customization to ensure high accuracy. Choosing such hyper-parameters is cumbersome as the pertinent search space grows exponentially with model layers. This paper introduces GeneCAI, a novel optimization method that automatically learns how to tune per-layer compression hyper-parameters. We devise a bijective translation scheme that encodes compressed DNNs to the genotype space. The optimality of each genotype is measured using a multi-objective score based on accuracy and number of floating point operations. We develop customized genetic operations to iteratively evolve the non-dominated solutions towards the optimal Pareto front, thus, capturing the optimal trade-off between model accuracy and complexity. GeneCAI optimization method is highly scalable and can achieve a near-linear performance boost on distributed multi-GPU platforms. Our extensive evaluations demonstrate that GeneCAI outperforms existing rule-based and reinforcement learning methods in DNN compression by finding models that lie on a better accuracy-complexity Pareto curve.

References in corpus (17)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
• Pruning Convolutional Neural Networks for Resource Efficient Inference
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Evolution Strategies as a Scalable Alternative to Reinforcement Learning
• Pruning Filters for Efficient ConvNets
• Learning both Weights and Connections for Efficient Neural Networks
• Large-Scale Evolution of Image Classifiers
• Channel Pruning for Accelerating Very Deep Neural Networks
• NetAdapt: Platform-Aware Neural Network Adaptation for Mobile Applications
• A novel channel pruning method for deep neural network compression
• Structured Probabilistic Pruning for Convolutional Neural Network Acceleration
• HAQ: Hardware-Aware Automated Quantization with Mixed Precision
• ReLeQ: A Reinforcement Learning Approach for Deep Quantization of Neural Networks
• CodeX: Bit-Flexible Encoding for Streaming-based FPGA Acceleration of DNNs
• SWNet: Small-World Neural Networks and Rapid Convergence

Cited by in corpus (2)

• A Survey on Evolutionary Neural Architecture Search
• HyperTune: Dynamic Hyperparameter Tuning For Efficient Distribution of DNN Training Over Heterogeneous Systems