An Analysis of Neural Language Modeling at Multiple Scales

Abstract

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.

References in corpus (9)

• Neural Architecture Search with Reinforcement Learning
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Regularizing and Optimizing LSTM Language Models
• Learning to Generate Reviews and Discovering Sentiment
• Hierarchical Multiscale Recurrent Neural Networks
• Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling
• Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations
• Revisiting Activation Regularization for Language RNNs
• Surprisal-Driven Zoneout

Cited by in corpus (43)

• Working Memory Connections for LSTM
• fairseq: A Fast, Extensible Toolkit for Sequence Modeling
• Mixed-precision deep learning based on computational memory
• Trellis Networks for Sequence Modeling
• Single Headed Attention RNN: Stop Thinking With Your Head
• A Survey of Deep Learning for Scientific Discovery
• DeLighT: Deep and Light-weight Transformer
• Augmenting Self-attention with Persistent Memory
• Discrete Flows: Invertible Generative Models of Discrete Data
• Dynamic Evaluation of Transformer Language Models
• Addressing Some Limitations of Transformers with Feedback Memory
• The Implicit and Explicit Regularization Effects of Dropout
• Exploring Self-Identified Counseling Expertise in Online Support Forums
• GitHub Typo Corpus: A Large-Scale Multilingual Dataset of Misspellings and Grammatical Errors
• Time-aware Large Kernel Convolutions
• Rethinking Full Connectivity in Recurrent Neural Networks
• DeFINE: DEep Factorized INput Token Embeddings for Neural Sequence Modeling
• Calibration, Entropy Rates, and Memory in Language Models
• Neural Polysynthetic Language Modelling
• ERNIE-Doc: A Retrospective Long-Document Modeling Transformer
• Low-Resource Language Modelling of South African Languages
• Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM
• Transfer Learning for British Sign Language Modelling
• Exploring the Vulnerability of Deep Neural Networks: A Study of Parameter Corruption
• Incorporating Word and Subword Units in Unsupervised Machine Translation Using Language Model Rescoring
• Improved memory in recurrent neural networks with sequential non-normal dynamics
• Progress and Tradeoffs in Neural Language Models
• Adaptive Pruning of Neural Language Models for Mobile Devices
• On Extending NLP Techniques from the Categorical to the Latent Space: KL Divergence, Zipf's Law, and Similarity Search
• Automated Source Code Generation and Auto-completion Using Deep Learning: Comparing and Discussing Current Language-Model-Related Approaches
• Efficient Contextual Representation Learning Without Softmax Layer
• Contextual Text Embeddings for Twi
• Meaning to Form: Measuring Systematicity as Information
• Character n-gram Embeddings to Improve RNN Language Models
• Character-based Surprisal as a Model of Reading Difficulty in the Presence of Error
• Learning Architectures from an Extended Search Space for Language Modeling
• Sparse Meta Networks for Sequential Adaptation and its Application to Adaptive Language Modelling
• Analysing Dropout and Compounding Errors in Neural Language Models
• Improving Neural Language Models by Segmenting, Attending, and Predicting the Future
• E.T.: Entity-Transformers. Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks
• Evaluating Computational Language Models with Scaling Properties of Natural Language
• Tabula nearly rasa: Probing the Linguistic Knowledge of Character-Level Neural Language Models Trained on Unsegmented Text
• Revisiting Simple Neural Probabilistic Language Models