On-FPGA Training with Ultra Memory Reduction: A Low-Precision Tensor Method
arXiv:2104.03420
Abstract
Various hardware accelerators have been developed for energy-efficient and real-time inference of neural networks on edge devices. However, most training is done on high-performance GPUs or servers, and the huge memory and computing costs prevent training neural networks on edge devices. This paper proposes a novel tensor-based training framework, which offers orders-of-magnitude memory reduction in the training process. We propose a novel rank-adaptive tensorized neural network model, and design a hardware-friendly low-precision algorithm to train this model. We present an FPGA accelerator to demonstrate the benefits of this training method on edge devices. Our preliminary FPGA implementation achieves speedup and energy reduction compared to embedded CPU, and memory reduction over a standard full-size training.
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms
- Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
- Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights
- Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition
- Structured Bayesian Pruning via Log-Normal Multiplicative Noise
- Tensorized Embedding Layers for Efficient Model Compression
- Compression and Interpretability of Deep Neural Networks via Tucker Tensor Layer: From First Principles to Tensor Valued Back-Propagation