3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

Abstract

The deep neural network (DNN) based AI applications on the edge require both low-cost computing platforms and high-quality services. However, the limited memory, computing resources, and power budget of the edge devices constrain the effectiveness of the DNN algorithms. Developing edge-oriented AI algorithms and implementations (e.g., accelerators) is challenging. In this paper, we summarize our recent efforts for efficient on-device AI development from three aspects, including both training and inference. First, we present on-device training with ultra-low memory usage. We propose a novel rank-adaptive tensor-based tensorized neural network model, which offers orders-of-magnitude memory reduction during training. Second, we introduce an ultra-low bitwidth quantization method for DNN model compression, achieving the state-of-the-art accuracy under the same compression ratio. Third, we introduce an ultra-low latency DNN accelerator design, practicing the software/hardware co-design methodology. This paper emphasizes the importance and efficacy of training, quantization and accelerator design, and calls for more research breakthroughs in the area for AI on the edge.

arXiv comment6 pages

References in corpus (14)

• Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
• Ternary Weight Networks
• Tensorizing Neural Networks
• Deep Learning Recommendation Model for Personalization and Recommendation Systems
• Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
• Hardware-oriented Approximation of Convolutional Neural Networks
• Compressing Recurrent Neural Network with Tensor Train
• Ultimate tensorization: compressing convolutional and FC layers alike
• Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM
• Tensorized Embedding Layers for Efficient Model Compression
• Compression and Interpretability of Deep Neural Networks via Tucker Tensor Layer: From First Principles to Tensor Valued Back-Propagation
• Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition
• Bayesian Tensorized Neural Networks with Automatic Rank Selection
• On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method