Von Mises-Fisher Loss for Training Sequence to Sequence Models with Continuous Outputs

Abstract

The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for replacing the softmax layer with a continuous embedding layer. Our primary innovations are a novel probabilistic loss, and a training and inference procedure in which we generate a probability distribution over pre-trained word embeddings, instead of a multinomial distribution over the vocabulary obtained via softmax. We evaluate this new class of sequence-to-sequence models with continuous outputs on the task of neural machine translation. We show that our models obtain upto 2.5x speed-up in training time while performing on par with the state-of-the-art models in terms of translation quality. These models are capable of handling very large vocabularies without compromising on translation quality. They also produce more meaningful errors than in the softmax-based models, as these errors typically lie in a subspace of the vector space of the reference translations.

Seventh International Conference on Learning Representations (ICLR 2019)

Cited by in corpus (22)

• An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models
• The Power Spherical distribution
• Sparse and Continuous Attention Mechanisms
• compare-mt: A Tool for Holistic Comparison of Language Generation Systems
• Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries
• Neural Machine Translation: A Review of Methods, Resources, and Tools
• Sparse Sequence-to-Sequence Models
• Multimodal Machine Translation with Embedding Prediction
• Plug and Play Autoencoders for Conditional Text Generation
• Discriminative Few-Shot Learning Based on Directional Statistics
• Efficient Contextual Representation Learning Without Softmax Layer
• Worse WER, but Better BLEU? Leveraging Word Embedding as Intermediate in Multitask End-to-End Speech Translation
• Controlled Text Generation as Continuous Optimization with Multiple Constraints
• On The Distribution of Penultimate Activations of Classification Networks
• Directional Deep Embedding and Appearance Learning for Fast Video Object Segmentation
• Multi-Facet Recommender Networks with Spherical Optimization
• Unsupervised Transfer Learning in Multilingual Neural Machine Translation with Cross-Lingual Word Embeddings
• Sparse Continuous Distributions and Fenchel-Young Losses
• Hierarchical Metadata-Aware Document Categorization under Weak Supervision
• Refine Neutrino Events Reconstruction with BEiT-3
• Regressing Word and Sentence Embeddings for Regularization of Neural Machine Translation
• Improving the Diversity of Unsupervised Paraphrasing with Embedding Outputs