Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

Abstract

Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• Recurrent Neural Network Regularization
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• A Theoretically Grounded Application of Dropout in Recurrent Neural Networks
• Pointer Sentinel Mixture Models

Cited by in corpus (86)

• Neural Architecture Search with Reinforcement Learning
• A Deep Reinforced Model for Abstractive Summarization
• Federated Learning for Mobile Keyboard Prediction
• Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
• How To Backdoor Federated Learning
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Efficient Neural Architecture Search via Parameter Sharing
• Regularizing and Optimizing LSTM Language Models
• Weighted Transformer Network for Machine Translation
• An Analysis of Neural Language Modeling at Multiple Scales
• Independently Recurrent Neural Network (IndRNN): Building A Longer and Deeper RNN
• FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network
• Fast Abstractive Summarization with Reinforce-Selected Sentence Rewriting
• Multiplicative LSTM for sequence modelling
• Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning
• Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networks
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS
• Breaking the Softmax Bottleneck: A High-Rank RNN Language Model
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• Single Headed Attention RNN: Stop Thinking With Your Head
• Adaptive Input Representations for Neural Language Modeling
• Tensorized Embedding Layers for Efficient Model Compression
• Language Models with Transformers
• Compression of Recurrent Neural Networks for Efficient Language Modeling
• Revisiting Activation Regularization for Language RNNs
• Towards Binary-Valued Gates for Robust LSTM Training
• Deep Independently Recurrent Neural Network (IndRNN)
• ProGen: Language Modeling for Protein Generation
• Deep Encoder, Shallow Decoder: Reevaluating Non-autoregressive Machine Translation
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation
• Tensor Decomposition for Model Reduction in Neural Networks: A Review
• Tied & Reduced RNN-T Decoder
• Learning Visually-Grounded Semantics from Contrastive Adversarial Samples
• You May Not Need Attention
• Neural Language Modeling by Jointly Learning Syntax and Lexicon
• Noisin: Unbiased Regularization for Recurrent Neural Networks
• PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation
• Graph Convolutional Networks for Classification with a Structured Label Space
• Selfish Sparse RNN Training
• PowerNorm: Rethinking Batch Normalization in Transformers
• Neural Document Summarization by Jointly Learning to Score and Select Sentences
• Restricted Recurrent Neural Networks
• The Importance of Being Recurrent for Modeling Hierarchical Structure
• rTop-k: A Statistical Estimation Approach to Distributed SGD
• Fix your classifier: the marginal value of training the last weight layer
• Rethinking embedding coupling in pre-trained language models
• Efficient Neural Architecture Search via Proximal Iterations
• Regularizing activations in neural networks via distribution matching with the Wasserstein metric
• Breaking the Activation Function Bottleneck through Adaptive Parameterization
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• Language Modeling with Sparse Product of Sememe Experts
• When FastText Pays Attention: Efficient Estimation of Word Representations using Constrained Positional Weighting
• Progress and Tradeoffs in Neural Language Models
• Adaptive Pruning of Neural Language Models for Mobile Devices
• Early Improving Recurrent Elastic Highway Network
• The Rediscovery Hypothesis: Language Models Need to Meet Linguistics
• Deep-speare: A Joint Neural Model of Poetic Language, Meter and Rhyme
• Transformer on a Diet
• Inducing Grammars with and for Neural Machine Translation
• Direct Output Connection for a High-Rank Language Model
• An Analysis of the Utility of Explicit Negative Examples to Improve the Syntactic Abilities of Neural Language Models
• Learning Less-Overlapping Representations
• Improving Human Text Comprehension through Semi-Markov CRF-based Neural Section Title Generation
• Low-Rank RNN Adaptation for Context-Aware Language Modeling
• Improving Context Aware Language Models
• Kite: Automatic speech recognition for unmanned aerial vehicles
• GILE: A Generalized Input-Label Embedding for Text Classification
• Towards one-shot learning for rare-word translation with external experts
• Spectral Pruning for Recurrent Neural Networks
• Assessing the Memory Ability of Recurrent Neural Networks
• Accelerated Reinforcement Learning for Sentence Generation by Vocabulary Prediction
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Analysing Dropout and Compounding Errors in Neural Language Models
• Applying Recent Innovations from NLP to MOOC Student Course Trajectory Modeling
• Ouroboros: On Accelerating Training of Transformer-Based Language Models
• Adaptive Noise Injection: A Structure-Expanding Regularization for RNN
• An Embedded Deep Learning based Word Prediction
• Subword Language Model for Query Auto-Completion
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• Improving OOV Detection and Resolution with External Language Models in Acoustic-to-Word ASR
• Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation
• Convex Aggregation for Opinion Summarization
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora
• Neural Bradley-Terry Rating: Quantifying Properties from Comparisons
• Learning Sparse Structured Ensembles with SG-MCMC and Network Pruning