The Natural Language Decathlon: Multitask Learning as Question Answering

Abstract

Deep learning has improved performance on many natural language processing (NLP) tasks individually. However, general NLP models cannot emerge within a paradigm that focuses on the particularities of a single metric, dataset, and task. We introduce the Natural Language Decathlon (decaNLP), a challenge that spans ten tasks: question answering, machine translation, summarization, natural language inference, sentiment analysis, semantic role labeling, zero-shot relation extraction, goal-oriented dialogue, semantic parsing, and commonsense pronoun resolution. We cast all tasks as question answering over a context. Furthermore, we present a new Multitask Question Answering Network (MQAN) jointly learns all tasks in decaNLP without any task-specific modules or parameters in the multitask setting. MQAN shows improvements in transfer learning for machine translation and named entity recognition, domain adaptation for sentiment analysis and natural language inference, and zero-shot capabilities for text classification. We demonstrate that the MQAN's multi-pointer-generator decoder is key to this success and performance further improves with an anti-curriculum training strategy. Though designed for decaNLP, MQAN also achieves state of the art results on the WikiSQL semantic parsing task in the single-task setting. We also release code for procuring and processing data, training and evaluating models, and reproducing all experiments for decaNLP.

References in corpus (17)

• Sequence to Sequence Learning with Neural Networks
• Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation
• Natural Language Processing (almost) from Scratch
• Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning
• PathNet: Evolution Channels Gradient Descent in Super Neural Networks
• Layer Normalization
• Machine Comprehension Using Match-LSTM and Answer Pointer
• QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
• One Model To Learn Them All
• DCN+: Mixed Objective and Deep Residual Coattention for Question Answering
• Global-Locally Self-Attentive Dialogue State Tracker
• Distance-based Self-Attention Network for Natural Language Inference
• MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension
• Coarse-to-Fine Decoding for Neural Semantic Parsing
• Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
• The University of Edinburgh's Neural MT Systems for WMT17
• Phase Conductor on Multi-layered Attentions for Machine Comprehension

Cited by in corpus (138)

• Learning Transferable Visual Models From Natural Language Supervision
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Language Models are Few-Shot Learners
• SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
• Multi-Task Learning for Dense Prediction Tasks: A Survey
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• mT5: A massively multilingual pre-trained text-to-text transformer
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Multi-Task Learning with Deep Neural Networks: A Survey
• BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning
• Unifying Vision-and-Language Tasks via Text Generation
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• QA Dataset Explosion: A Taxonomy of NLP Resources for Question Answering and Reading Comprehension
• Structured Neural Summarization
• Learning and Evaluating General Linguistic Intelligence
• A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Gradient Surgery for Multi-Task Learning
• DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue
• Rethinking Search: Making Domain Experts out of Dilettantes
• Paradigm Shift in Natural Language Processing
• Multi-task learning for natural language processing in the 2020s: where are we going?
• KLUE: Korean Language Understanding Evaluation
• Zero-shot Text Classification With Generative Language Models
• Neural Abstractive Text Summarization with Sequence-to-Sequence Models
• IncSQL: Training Incremental Text-to-SQL Parsers with Non-Deterministic Oracles
• The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics
• A Unified MRC Framework for Named Entity Recognition
• RussianSuperGLUE: A Russian Language Understanding Evaluation Benchmark
• Machine Reading Comprehension: The Role of Contextualized Language Models and Beyond
• Entity-Relation Extraction as Multi-Turn Question Answering
• Understanding and Improving Information Transfer in Multi-Task Learning
• Explaining Question Answering Models through Text Generation
• How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG
• Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems
• Identifying Generalization Properties in Neural Networks
• Learning to summarize from human feedback
• 12-in-1: Multi-Task Vision and Language Representation Learning
• Structured Prediction as Translation between Augmented Natural Languages
• MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale
• LFPT5: A Unified Framework for Lifelong Few-shot Language Learning Based on Prompt Tuning of T5
• Evaluating Language Model Finetuning Techniques for Low-resource Languages
• Dice Loss for Data-imbalanced NLP Tasks
• Localization of Fake News Detection via Multitask Transfer Learning
• Description Based Text Classification with Reinforcement Learning
• Explicit Contextual Semantics for Text Comprehension
• Unifying Question Answering, Text Classification, and Regression via Span Extraction
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Attention over Parameters for Dialogue Systems
• EntQA: Entity Linking as Question Answering
• Event Extraction by Answering (Almost) Natural Questions
• A Comparative Study of Transformer-Based Language Models on Extractive Question Answering
• Coreference Resolution as Query-based Span Prediction
• Question Answering is a Format; When is it Useful?
• Probing and Fine-tuning Reading Comprehension Models for Few-shot Event Extraction
• Pretrained AI Models: Performativity, Mobility, and Change
• Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task
• TableQA: a Large-Scale Chinese Text-to-SQL Dataset for Table-Aware SQL Generation
• Simple and Effective Curriculum Pointer-Generator Networks for Reading Comprehension over Long Narratives
• Meta-Learning with Sparse Experience Replay for Lifelong Language Learning
• Knowledge Graph-based Question Answering with Electronic Health Records
• Using a thousand optimization tasks to learn hyperparameter search strategies
• Universal Semi-Supervised Semantic Segmentation
• Relation Classification as Two-way Span-Prediction
• Utility is in the Eye of the User: A Critique of NLP Leaderboards
• An Empirical Comparison on Imitation Learning and Reinforcement Learning for Paraphrase Generation
• Weakly-supervised Multi-task Learning for Multimodal Affect Recognition
• Towards Question Format Independent Numerical Reasoning: A Set of Prerequisite Tasks
• Automatic Construction of Evaluation Suites for Natural Language Generation Datasets
• Multi-style Generative Reading Comprehension
• Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning
• Exploring and Predicting Transferability across NLP Tasks
• Graph Meta Learning via Local Subgraphs
• Dynamic Hybrid Relation Network for Cross-Domain Context-Dependent Semantic Parsing
• Beam Search with Bidirectional Strategies for Neural Response Generation
• Learning Universal Graph Neural Network Embeddings With Aid Of Transfer Learning
• Biomedical named entity recognition using BERT in the machine reading comprehension framework
• GLGE: A New General Language Generation Evaluation Benchmark
• Text-to-SQL Generation for Question Answering on Electronic Medical Records
• Task Selection Policies for Multitask Learning
• All-in-One Image-Grounded Conversational Agents
• Structuring Latent Spaces for Stylized Response Generation
• Symbolic inductive bias for visually grounded learning of spoken language
• A Supervised Word Alignment Method based on Cross-Language Span Prediction using Multilingual BERT
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• Deep Learning Applied to Chest X-Rays: Exploiting and Preventing Shortcuts
• SpeechNet: A Universal Modularized Model for Speech Processing Tasks
• Towards General Purpose Vision Systems
• Compositional Generalization via Semantic Tagging
• A Brief Review of Deep Multi-task Learning and Auxiliary Task Learning
• Weighted Training for Cross-Task Learning
• Speech Representation Learning Through Self-supervised Pretraining And Multi-task Finetuning
• Zero-shot Generalization in Dialog State Tracking through Generative Question Answering
• Learning Functions to Study the Benefit of Multitask Learning
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• Lifelong Language Knowledge Distillation
• Knowledge Guided Named Entity Recognition for BioMedical Text
• AutoSeM: Automatic Task Selection and Mixing in Multi-Task Learning
• Modelling Latent Skills for Multitask Language Generation
• An MRC Framework for Semantic Role Labeling
• Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start
• Improving Generalization by Incorporating Coverage in Natural Language Inference
• Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining
• Challenges and Prospects in Vision and Language Research
• Efficient Meta Lifelong-Learning with Limited Memory
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
• Multi-Instance Learning for End-to-End Knowledge Base Question Answering
• MATINF: A Jointly Labeled Large-Scale Dataset for Classification, Question Answering and Summarization
• CLEVA-Compass: A Continual Learning EValuation Assessment Compass to Promote Research Transparency and Comparability
• Continual Learning in Task-Oriented Dialogue Systems
• PolyViT: Co-training Vision Transformers on Images, Videos and Audio
• A Hybrid Semantic Parsing Approach for Tabular Data Analysis
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Forget Me Not: Reducing Catastrophic Forgetting for Domain Adaptation in Reading Comprehension
• Problems and Countermeasures in Natural Language Processing Evaluation
• Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collections
• Bridging Anaphora Resolution as Question Answering
• Datasets: A Community Library for Natural Language Processing
• Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning
• Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems
• Query-Based Named Entity Recognition
• Transferable Natural Language Interface to Structured Queries aided by Adversarial Generation
• WaLDORf: Wasteless Language-model Distillation On Reading-comprehension
• Should We Be Pre-training? An Argument for End-task Aware Training as an Alternative
• Translating Natural Language to SQL using Pointer-Generator Networks and How Decoding Order Matters
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
• Hierarchical Multi Task Learning with Subword Contextual Embeddings for Languages with Rich Morphology
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
• Balancing Average and Worst-case Accuracy in Multitask Learning
• FewshotQA: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models
• Toward Human-Level Artificial Intelligence
• Neural Duplicate Question Detection without Labeled Training Data
• MTLHealth: A Deep Learning System for Detecting Disturbing Content in Student Essays
• Doc2Dict: Information Extraction as Text Generation
• A High-Quality Multilingual Dataset for Structured Documentation Translation
• Syntactic Question Abstraction and Retrieval for Data-Scarce Semantic Parsing