Lightweight Convolutional Representations for On-Device Natural Language Processing
arXiv:2002.01535
Abstract
The increasing computational and memory complexities of deep neural networks have made it difficult to deploy them on low-resource electronic devices (e.g., mobile phones, tablets, wearables). Practitioners have developed numerous model compression methods to address these concerns, but few have condensed input representations themselves. In this work, we propose a fast, accurate, and lightweight convolutional representation that can be swapped into any neural model and compressed significantly (up to 32x) with a negligible reduction in performance. In addition, we show gains over recurrent representations when considering resource-centric metrics (e.g., model file size, latency, memory usage) on a Samsung Galaxy S9.
Accepted to MLSys 2020
References in corpus (10)
- Distilling the Knowledge in a Neural Network
- Deep Residual Learning for Image Recognition
- Gaussian Error Linear Units (GELUs)
- Convolutional Sequence to Sequence Learning
- Empirical Evaluation of Rectified Activations in Convolutional Network
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Compressing Deep Convolutional Networks using Vector Quantization
- Fixed Point Quantization of Deep Convolutional Networks
- The State of Sparsity in Deep Neural Networks
- Accelerator-Aware Pruning for Convolutional Neural Networks