Exploring the Limits of Language Modeling

Abstract

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

References in corpus (10)

• Sequence to Sequence Learning with Neural Networks
• On the difficulty of training Recurrent Neural Networks
• Recurrent Neural Network Regularization
• A Neural Conversational Model
• Training Very Deep Networks
• Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation
• Addressing the Rare Word Problem in Neural Machine Translation
• Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs
• Document Context Language Models
• Larger-Context Language Modelling

Cited by in corpus (279)

• TensorFlow: A system for large-scale machine learning
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence
• Cross-lingual Language Model Pretraining
• Language Modeling with Gated Convolutional Networks
• Federated Learning for Mobile Keyboard Prediction
• Mixed Precision Training
• Contrastive Representation Learning: A Framework and Review
• Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization
• Density estimation using Real NVP
• Revisiting Distributed Synchronous SGD
• Deep Bayesian Active Learning with Image Data
• Generating Long Sequences with Sparse Transformers
• Deep Learning Scaling is Predictable, Empirically
• Learning to Generate Reviews and Discovering Sentiment
• Recent Advances in Convolutional Neural Networks
• A Simple Method for Commonsense Reasoning
• CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge
• What do Neural Machine Translation Models Learn about Morphology?
• Self-supervised Co-training for Video Representation Learning
• Distilling a Neural Network Into a Soft Decision Tree
• Learning Video Representations using Contrastive Bidirectional Transformer
• Device Placement Optimization with Reinforcement Learning
• Spectral Norm Regularization for Improving the Generalizability of Deep Learning
• Neural Voice Cloning with a Few Samples
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Measuring the Effects of Data Parallelism on Neural Network Training
• Fault Detection and Identification using Bayesian Recurrent Neural Networks
• Supporting Very Large Models using Automatic Dataflow Graph Partitioning
• Learning for Video Compression with Recurrent Auto-Encoder and Recurrent Probability Model
• Big Code != Big Vocabulary: Open-Vocabulary Models for Source Code
• Bayesian Recurrent Neural Networks
• Exploring Sparsity in Recurrent Neural Networks
• Efficient softmax approximation for GPUs
• Thinking Fast and Slow: Efficient Text-to-Visual Retrieval with Transformers
• Network Traffic Anomaly Detection Using Recurrent Neural Networks
• DynaBERT: Dynamic BERT with Adaptive Width and Depth
• Scaling Distributed Machine Learning with In-Network Aggregation
• Recent Advances in Deep Learning: An Overview
• A Neural Knowledge Language Model
• Sequence-Level Knowledge Distillation
• Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP
• Latent Predictor Networks for Code Generation
• Factorization tricks for LSTM networks
• Block-Sparse Recurrent Neural Networks
• LSTM-Based System-Call Language Modeling and Robust Ensemble Method for Designing Host-Based Intrusion Detection Systems
• Selection via Proxy: Efficient Data Selection for Deep Learning
• Defending Against Neural Fake News
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling
• Language Generation with Recurrent Generative Adversarial Networks without Pre-training
• Neural Models for Information Retrieval
• Charformer: Fast Character Transformers via Gradient-based Subword Tokenization
• Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs
• An Online Multilingual Hate speech Recognition System
• Optimizing Performance of Recurrent Neural Networks on GPUs
• So Cloze yet so Far: N400 Amplitude is Better Predicted by Distributional Information than Human Predictability Judgements
• OpenAttack: An Open-source Textual Adversarial Attack Toolkit
• Gender Bias in Neural Natural Language Processing
• Generating Focussed Molecule Libraries for Drug Discovery with Recurrent Neural Networks
• End-to-End Retrieval in Continuous Space
• Automatic Rule Extraction from Long Short Term Memory Networks
• Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
• TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP
• Detecting Cross-Modal Inconsistency to Defend Against Neural Fake News
• Reading Scene Text with Attention Convolutional Sequence Modeling
• Adaptive Input Representations for Neural Language Modeling
• Training LSTM Networks with Resistive Cross-Point Devices
• CHiVE: Varying Prosody in Speech Synthesis with a Linguistically Driven Dynamic Hierarchical Conditional Variational Network
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Mesh-TensorFlow: Deep Learning for Supercomputers
• A Hybrid Convolutional Variational Autoencoder for Text Generation
• Augmenting Self-attention with Persistent Memory
• Gmail Smart Compose: Real-Time Assisted Writing
• Charagram: Embedding Words and Sentences via Character n-grams
• Gated Word-Character Recurrent Language Model
• RNNs as psycholinguistic subjects: Syntactic state and grammatical dependency
• Unifying Human and Statistical Evaluation for Natural Language Generation
• Towards Binary-Valued Gates for Robust LSTM Training
• SyntaxNet Models for the CoNLL 2017 Shared Task
• An Updated Duet Model for Passage Re-ranking
• Neural Text Generation: A Practical Guide
• N-gram Language Modeling using Recurrent Neural Network Estimation
• Cerebro: Static Subsuming Mutant Selection
• Montage: A Neural Network Language Model-Guided JavaScript Engine Fuzzer
• Recurrent Additive Networks
• Tensor Decomposition for Model Reduction in Neural Networks: A Review
• Program Synthesis with Large Language Models
• CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters
• Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies
• A Systematic Assessment of Syntactic Generalization in Neural Language Models
• Relational recurrent neural networks
• GDP: Generalized Device Placement for Dataflow Graphs
• Modeling Vocabulary for Big Code Machine Learning
• Named Entity Recognition with stack residual LSTM and trainable bias decoding
• CharacterBERT and Self-Teaching for Improving the Robustness of Dense Retrievers on Queries with Typos
• Generating Natural Language Explanations for Visual Question Answering using Scene Graphs and Visual Attention
• Probabilistic Deep Learning to Quantify Uncertainty in Air Quality Forecasting
• An Empirical Study of Training End-to-End Vision-and-Language Transformers
• Affect-LM: A Neural Language Model for Customizable Affective Text Generation
• Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates
• Fast Parametric Learning with Activation Memorization
• Sequence Model Design for Code Completion in the Modern IDE
• Combining Convolution and Recursive Neural Networks for Sentiment Analysis
• Malicious Code Detection: Run Trace Output Analysis by LSTM
• Implicit Dimension Identification in User-Generated Text with LSTM Networks
• Robust Lexical Features for Improved Neural Network Named-Entity Recognition
• Using Similarity Measures to Select Pretraining Data for NER
• GNN-LM: Language Modeling based on Global Contexts via GNN
• MinimalRNN: Toward More Interpretable and Trainable Recurrent Neural Networks
• Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism
• Character-Word LSTM Language Models
• SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders
• Unsupervised Pretraining for Sequence to Sequence Learning
• Towards Robust Toxic Content Classification
• Self-Knowledge Distillation with Progressive Refinement of Targets
• Compositionality decomposed: how do neural networks generalise?
• Learning in Text Streams: Discovery and Disambiguation of Entity and Relation Instances
• How to Train Your Energy-Based Model for Regression
• What Syntactic Structures block Dependencies in RNN Language Models?
• Blackbox meets blackbox: Representational Similarity and Stability Analysis of Neural Language Models and Brains
• Different kinds of cognitive plausibility: why are transformers better than RNNs at predicting N400 amplitude?
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Calibration, Entropy Rates, and Memory in Language Models
• What Looks Good with my Sofa: Multimodal Search Engine for Interior Design
• Measuring scheduling efficiency of RNNs for NLP applications
• Generating Steganographic Text with LSTMs
• A Neural Language Model for Dynamically Representing the Meanings of Unknown Words and Entities in a Discourse
• Learning to Match Using Local and Distributed Representations of Text for Web Search
• Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State
• End-to-end deep meta modelling to calibrate and optimize energy consumption and comfort
• The Impact of LoRA Adapters on LLMs for Clinical Text Classification Under Computational and Data Constraints
• A Comprehensive Survey of Deep Learning for Image Captioning
• Boosting Distributed Machine Learning Training Through Loss-tolerant Transmission Protocol
• Unbounded cache model for online language modeling with open vocabulary
• Neural Classification of Malicious Scripts: A study with JavaScript and VBScript
• Self-Adversarial Learning with Comparative Discrimination for Text Generation
• Speech Separation with Pretrained Frontend to Minimize Domain Mismatch
• The Woman Worked as a Babysitter: On Biases in Language Generation
• Using Deep Networks and Transfer Learning to Address Disinformation
• Reference-Aware Language Models
• The Good, the Bad, and the Missing: Neural Code Generation for Machine Learning Tasks
• Why Build an Assistant in Minecraft?
• Learning Sentence Representation with Guidance of Human Attention
• Language Models as Emotional Classifiers for Textual Conversations
• Noise-Contrastive Estimation for Multivariate Point Processes
• Reinforcement Learning Based Text Style Transfer without Parallel Training Corpus
• Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiency
• Are Character-level Translations Worth the Wait? Comparing ByT5 and mT5 for Machine Translation
• Local word statistics affect reading times independently of surprisal
• Lightweight Adaptive Mixture of Neural and N-gram Language Models
• Self-Normalization Properties of Language Modeling
• Likelihood Ratios for Out-of-Distribution Detection
• PT-CoDE: Pre-trained Context-Dependent Encoder for Utterance-level Emotion Recognition
• Building Language Models for Text with Named Entities
• Compound Probabilistic Context-Free Grammars for Grammar Induction
• Character-Level Language Modeling with Hierarchical Recurrent Neural Networks
• Accelerated Training for CNN Distributed Deep Learning through Automatic Resource-Aware Layer Placement
• Unsupervised and Efficient Vocabulary Expansion for Recurrent Neural Network Language Models in ASR
• Beyond Human-Level Accuracy: Computational Challenges in Deep Learning
• CNN Is All You Need
• Exploring the Syntactic Abilities of RNNs with Multi-task Learning
• Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding
• Long Short-Term Attention
• Comparative Evaluation of Pretrained Transfer Learning Models on Automatic Short Answer Grading
• Recurrent Neural Networks as Weighted Language Recognizers
• Dissecting Contextual Word Embeddings: Architecture and Representation
• EmbRace: Accelerating Sparse Communication for Distributed Training of NLP Neural Networks
• DeepReduce: A Sparse-tensor Communication Framework for Distributed Deep Learning
• Efficient Contextualized Representation: Language Model Pruning for Sequence Labeling
• Low-rank passthrough neural networks
• Optimizing Multi-GPU Parallelization Strategies for Deep Learning Training
• Say What? Collaborative Pop Lyric Generation Using Multitask Transfer Learning
• Improving Neural Machine Translation with Pre-trained Representation
• Compressing Gradient Optimizers via Count-Sketches
• Assessing Language Models with Scaling Properties
• Polite Dialogue Generation Without Parallel Data
• TAPAS: Two-pass Approximate Adaptive Sampling for Softmax
• Cross-Utterance Language Models with Acoustic Error Sampling
• Judge the Judges: A Large-Scale Evaluation Study of Neural Language Models for Online Review Generation
• Anchor & Transform: Learning Sparse Embeddings for Large Vocabularies
• GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinking
• Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data
• Learning to Explain: Answering Why-Questions via Rephrasing
• End-to-end deep metamodeling to calibrate and optimize energy loads
• Structural Supervision Improves Learning of Non-Local Grammatical Dependencies
• A Character-Word Compositional Neural Language Model for Finnish
• Read, Tag, and Parse All at Once, or Fully-neural Dependency Parsing
• Unsupervised Transfer Learning for Spoken Language Understanding in Intelligent Agents
• TopicEq: A Joint Topic and Mathematical Equation Model for Scientific Texts
• emLam -- a Hungarian Language Modeling baseline
• Contextual Text Embeddings for Twi
• Self-organized Hierarchical Softmax
• Efficient Contextual Representation Learning Without Softmax Layer
• Analysis Methods in Neural Language Processing: A Survey
• Recurrent Neural Networks with Pre-trained Language Model Embedding for Slot Filling Task
• Variational Hyper RNN for Sequence Modeling
• A Survey of Techniques All Classifiers Can Learn from Deep Networks: Models, Optimizations, and Regularization
• Combating Disinformation in a Social Media Age
• IntenT5: Search Result Diversification using Causal Language Models
• A Survey on Low-Resource Neural Machine Translation
• Compressing LSTM Networks by Matrix Product Operators
• Predictively Encoded Graph Convolutional Network for Noise-Robust Skeleton-based Action Recognition
• TTPP: Temporal Transformer with Progressive Prediction for Efficient Action Anticipation
• Representation of Constituents in Neural Language Models: Coordination Phrase as a Case Study
• Accelerating Large Scale Knowledge Distillation via Dynamic Importance Sampling
• Neural Random Projections for Language Modelling
• Syntax-Aware Language Modeling with Recurrent Neural Networks
• Large-scale Cloze Test Dataset Created by Teachers
• Fast Single-Class Classification and the Principle of Logit Separation
• Repurposing TREC-COVID Annotations to Answer the Key Questions of CORD-19
• Towards A Multi-agent System for Online Hate Speech Detection
• Few-Shot Object Recognition from Machine-Labeled Web Images
• Multi-view Subword Regularization
• Evaluating Neural Word Embeddings for Sanskrit
• Domain-aware Neural Language Models for Speech Recognition
• Character n-gram Embeddings to Improve RNN Language Models
• AMSI-Based Detection of Malicious PowerShell Code Using Contextual Embeddings
• Efficient Large-Scale Domain Classification with Personalized Attention
• Survey: Transformer based Video-Language Pre-training
• Do RNNs learn human-like abstract word order preferences?
• UIBert: Learning Generic Multimodal Representations for UI Understanding
• DSKG: A Deep Sequential Model for Knowledge Graph Completion
• Training Feedforward Neural Networks with Standard Logistic Activations is Feasible
• Deep Learning in Memristive Nanowire Networks
• Towards Near-imperceptible Steganographic Text
• Masked ELMo: An evolution of ELMo towards fully contextual RNN language models
• How well does surprisal explain N400 amplitude under different experimental conditions?
• Energy-Inspired Models: Learning with Sampler-Induced Distributions
• Lattice Rescoring Strategies for Long Short Term Memory Language Models in Speech Recognition
• Cross-Lingual Transfer for Distantly Supervised and Low-resources Indonesian NER
• Multi-task Learning for Financial Forecasting
• WikiAtomicEdits: A Multilingual Corpus of Wikipedia Edits for Modeling Language and Discourse
• PaLM: A Hybrid Parser and Language Model
• Multimodal Embeddings from Language Models
• Character-Level Neural Translation for Multilingual Media Monitoring in the SUMMA Project
• Authorship Attribution in Bangla literature using Character-level CNN
• Conditional Generators of Words Definitions
• Leveraging Sequence Embedding and Convolutional Neural Network for Protein Function Prediction
• Recurrent Neural Network from Adder's Perspective: Carry-lookahead RNN
• Sampling-free Uncertainty Estimation in Gated Recurrent Units with Exponential Families
• Mixture of Expert/Imitator Networks: Scalable Semi-supervised Learning Framework
• PMI Matrix Approximations with Applications to Neural Language Modeling
• Do Transformers Need Deep Long-Range Memory
• Generative network complex (GNC) for drug discovery
• Pretrained language model transfer on neural named entity recognition in Indonesian conversational texts
• Unsupervised Deep Structured Semantic Models for Commonsense Reasoning
• Language Modeling with Highway LSTM
• Persistence pays off: Paying Attention to What the LSTM Gating Mechanism Persists
• A dependency look at the reality of constituency
• Learning Proposals for Practical Energy-Based Regression
• Identifiable Energy-based Representations: An Application to Estimating Heterogeneous Causal Effects
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories
• Linguistic Typology Features from Text: Inferring the Sparse Features of World Atlas of Language Structures
• Multi-Perspective Context Aggregation for Semi-supervised Cloze-style Reading Comprehension
• On Sampling-Based Training Criteria for Neural Language Modeling
• Near-imperceptible Neural Linguistic Steganography via Self-Adjusting Arithmetic Coding
• Vulgaris: Analysis of a Corpus for Middle-Age Varieties of Italian Language
• Restricted Recurrent Neural Tensor Networks: Exploiting Word Frequency and Compositionality
• Learning neural trans-dimensional random field language models with noise-contrastive estimation
• Shareable Representations for Search Query Understanding
• Transferable Graph Optimizers for ML Compilers
• Supervised Sentiment Classification with CNNs for Diverse SE Datasets
• A Generalized Language Model in Tensor Space
• Securing Voice-driven Interfaces against Fake (Cloned) Audio Attacks
• Meta-Embedding as Auxiliary Task Regularization
• Large Margin Neural Language Model
• A Random Gossip BMUF Process for Neural Language Modeling
• On Multilingual Training of Neural Dependency Parsers
• Investigating Alternatives to the Root Mean Square for Adaptive Gradient Methods
• Pre-Training Transformers as Energy-Based Cloze Models
• SAM: Semantic Attribute Modulation for Language Modeling and Style Variation
• Understood in Translation, Transformers for Domain Understanding
• Adversarial Machine Learning in Text Analysis and Generation
• Real-time Neural-based Input Method
• HinFlair: pre-trained contextual string embeddings for pos tagging and text classification in the Hindi language
• A Survey on Green Deep Learning