Classes for Fast Maximum Entropy Training
arXiv:cs/0108006
Abstract
Maximum entropy models are considered by many to be one of the most promising avenues of language modeling research. Unfortunately, long training times make maximum entropy research difficult. We present a novel speedup technique: we change the form of the model to use classes. Our speedup works by creating two maximum entropy models, the first of which predicts the class of each word, and the second of which predicts the word itself. This factoring of the model leads to fewer non-zero indicator functions, and faster normalization, achieving speedups of up to a factor of 35 over one of the best previous techniques. It also results in typically slightly lower perplexities. The same trick can be used to speed training of other machine learning techniques, e.g. neural networks, applied to any problem with a large number of outputs, such as language modeling.
4 pages
Cited by in corpus (19)
- DyNet: The Dynamic Neural Network Toolkit
- Efficient softmax approximation for GPUs
- Neural Machine Translation and Sequence-to-sequence Models: A Tutorial
- Strategies for Training Large Vocabulary Neural Language Models
- Fast Parametric Learning with Activation Memorization
- Online normalizer calculation for softmax
- Slim Embedding Layers for Recurrent Neural Language Models
- A Low Dimensionality Representation for Language Variety Identification
- Accelerated Training for Massive Classification via Dynamic Class Selection
- Building Language Models for Text with Named Entities
- Language Modeling with Sparse Product of Sememe Experts
- TAPAS: Two-pass Approximate Adaptive Sampling for Softmax
- Improving Classification When a Class Hierarchy is Available Using a Hierarchy-Based Prior
- Graph Embeddings at Scale
- A Character-Word Compositional Neural Language Model for Finnish
- Self-organized Hierarchical Softmax
- Adaptive Correlated Monte Carlo for Contextual Categorical Sequence Generation
- Scalable Machine Translation in Memory Constrained Environments
- A Bit of Progress in Language Modeling