Large Margin Neural Language Model
arXiv:1808.08987
Abstract
We propose a large margin criterion for training neural language models. Conventionally, neural language models are trained by minimizing perplexity (PPL) on grammatical sentences. However, we demonstrate that PPL may not be the best metric to optimize in some tasks, and further propose a large margin formulation. The proposed method aims to enlarge the margin between the "good" and "bad" sentences in a task-specific sense. It is trained end-to-end and can be widely applied to tasks that involve re-scoring of generated text. Compared with minimum-PPL training, our method gains up to 1.1 WER reduction for speech recognition and 1.0 BLEU increase for machine translation.
arXiv comment9 pages. Accepted as a long paper in EMNLP2018
References in corpus (6)
- Recurrent Neural Network Regularization
- Exploring the Limits of Language Modeling
- On Using Monolingual Corpora in Neural Machine Translation
- EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decoding
- Sequence-to-Sequence Learning as Beam-Search Optimization
- Gram-CTC: Automatic Unit Selection and Target Decomposition for Sequence Labelling