Learning and Evaluating General Linguistic Intelligence

Abstract

We define general linguistic intelligence as the ability to reuse previously acquired knowledge about a language's lexicon, syntax, semantics, and pragmatic conventions to adapt to new tasks quickly. Using this definition, we analyze state-of-the-art natural language understanding models and conduct an extensive empirical investigation to evaluate them against these criteria through a series of experiments that assess the task-independence of the knowledge being acquired by the learning process. In addition to task performance, we propose a new evaluation metric based on an online encoding of the test data that quantifies how quickly an existing agent (model) learns a new task. Our results show that while the field has made impressive progress in terms of model architectures that generalize to many tasks, these models still require a lot of in-domain training examples (e.g., for fine tuning, training task-specific modules), and are prone to catastrophic forgetting. Moreover, we find that far from solving general tasks (e.g., document question answering), our models are overfitting to the quirks of particular datasets (e.g., SQuAD). We discuss missing components and conjecture on how to make progress toward general linguistic intelligence.

References in corpus (9)

• Overcoming catastrophic forgetting in neural networks
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Achieving Human Parity on Automatic Chinese to English News Translation
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• On First-Order Meta-Learning Algorithms
• Semi-supervised Sequence Learning
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Improving Neural Language Models with a Continuous Cache
• The Description Length of Deep Learning Models

Cited by in corpus (42)

• Learning Transferable Visual Models From Natural Language Supervision
• Language Models are Few-Shot Learners
• On the Opportunities and Risks of Foundation Models
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Thieves on Sesame Street! Model Extraction of BERT-based APIs
• A Mutual Information Maximization Perspective of Language Representation Learning
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Learning from others' mistakes: Avoiding dataset biases without modeling them
• What do Models Learn from Question Answering Datasets?
• Neural Language Generation: Formulation, Methods, and Evaluation
• The Effect of Natural Distribution Shift on Question Answering Models
• Question Answering is a Format; When is it Useful?
• MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
• Language Tasks and Language Games: On Methodology in Current Natural Language Processing Research
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning
• Syntactic Data Augmentation Increases Robustness to Inference Heuristics
• Multi-style Generative Reading Comprehension
• Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling
• Evaluating representations by the complexity of learning low-loss predictors
• Data-Efficient Pretraining via Contrastive Self-Supervision
• Unsupervised Domain Adaptation of Contextualized Embeddings for Sequence Labeling
• BERT in Plutarch's Shadows
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems
• Improving Generalization by Incorporating Coverage in Natural Language Inference
• Continual Learning for Text Classification with Information Disentanglement Based Regularization
• Efficient Meta Lifelong-Learning with Limited Memory
• Automatic Validation of Textual Attribute Values in E-commerce Catalog by Learning with Limited Labeled Data
• Modeling Disclosive Transparency in NLP Application Descriptions
• Unsupervised Natural Question Answering with a Small Model
• Improving the Robustness of QA Models to Challenge Sets with Variational Question-Answer Pair Generation
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
• A Closer Look at Few-Shot Crosslingual Transfer: The Choice of Shots Matters
• Meta-Learning for Natural Language Understanding under Continual Learning Framework
• Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP
• Can Question Generation Debias Question Answering Models? A Case Study on Question-Context Lexical Overlap
• Kronecker Factorization for Preventing Catastrophic Forgetting in Large-scale Medical Entity Linking
• Bag-of-Vectors Autoencoders for Unsupervised Conditional Text Generation
• STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
• CALM: Continuous Adaptive Learning for Language Modeling
• Measuring Information Transfer in Neural Networks
• An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering