Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
arXiv:1804.07827
Abstract
Many efforts have been made to facilitate natural language processing tasks with pre-trained language models (LMs), and brought significant improvements to various applications. To fully leverage the nearly unlimited corpora and capture linguistic information of multifarious levels, large-size LMs are required; but for a specific task, only parts of these information are useful. Such large-sized LMs, even in the inference stage, may cause heavy computation workloads, making them too time-consuming for large-scale applications. Here we propose to compress bulky LMs while preserving useful information with regard to a specific task. As different layers of the model keep different information, we develop a layer selection method for model pruning using sparsity-inducing regularization. By introducing the dense connectivity, we can detach any layer without affecting others, and stretch shallow and wide LMs to be deep and narrow. In model training, LMs are learned with layer-wise dropouts for better robustness. Experiments on two benchmark datasets demonstrate the effectiveness of our method.
arXiv commentEMNLP 2018
References in corpus (11)
- Adam: A Method for Stochastic Optimization
- Distilling the Knowledge in a Neural Network
- Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1
- FitNets: Hints for Thin Deep Nets
- Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations
- Exploring the Limits of Language Modeling
- Learning Structured Sparsity in Deep Neural Networks
- Learning to Generate Reviews and Discovering Sentiment
- Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
- On the State of the Art of Evaluation in Neural Language Models
- Slim Embedding Layers for Recurrent Neural Language Models