Efficient Contextual Representation Learning Without Softmax Layer
arXiv:1902.11269
Abstract
Contextual representation models have achieved great success in improving various downstream tasks. However, these language-model-based encoders are difficult to train due to the large parameter sizes and high computational complexity. By carefully examining the training procedure, we find that the softmax layer (the output layer) causes significant inefficiency due to the large vocabulary size. Therefore, we redesign the learning objective and propose an efficient framework for training contextual representation models. Specifically, the proposed approach bypasses the softmax layer by performing language modeling with dimension reduction, and allows the models to leverage pre-trained word embeddings. Our framework reduces the time spent on the output layer to a negligible level, eliminates almost all the trainable parameters of the softmax layer and performs language modeling without truncating the vocabulary. When applied to ELMo, our method achieves a 4 times speedup and eliminates 80% trainable parameters while achieving competitive performance on downstream tasks.
Work in progress
References in corpus (13)
- Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
- Natural Language Processing (almost) from Scratch
- Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- Enhanced LSTM for Natural Language Inference
- cuDNN: Efficient Primitives for Deep Learning
- Character-Aware Neural Language Models
- Exploring the Limits of Language Modeling
- Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition
- Quasi-Recurrent Neural Networks
- A Fast and Simple Algorithm for Training Neural Probabilistic Language Models
- An Analysis of Neural Language Modeling at Multiple Scales
- Evaluation of sentence embeddings in downstream and linguistic probing tasks
- Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs