Multi-Task Deep Neural Networks for Natural Language Understanding

Abstract

In this paper, we present a Multi-Task Deep Neural Network (MT-DNN) for learning representations across multiple natural language understanding (NLU) tasks. MT-DNN not only leverages large amounts of cross-task data, but also benefits from a regularization effect that leads to more general representations in order to adapt to new tasks and domains. MT-DNN extends the model proposed in Liu et al. (2015) by incorporating a pre-trained bidirectional transformer language model, known as BERT (Devlin et al., 2018). MT-DNN obtains new state-of-the-art results on ten NLU tasks, including SNLI, SciTail, and eight out of nine GLUE tasks, pushing the GLUE benchmark to 82.7% (2.2% absolute improvement). We also demonstrate using the SNLI and SciTail datasets that the representations learned by MT-DNN allow domain adaptation with substantially fewer in-domain labels than the pre-trained BERT representations. The code and pre-trained models are publicly available at https://github.com/namisan/mt-dnn.

10 pages, 2 figures and 5 tables; Accepted by ACL 2019

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Natural Language Processing (almost) from Scratch
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• A Survey on Multi-Task Learning
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Stochastic Answer Networks for Natural Language Inference
• Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension

Cited by in corpus (91)

• Language Models are Few-Shot Learners
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Unified Language Model Pre-training for Natural Language Understanding and Generation
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• ERNIE: Enhanced Representation through Knowledge Integration
• Multi-Task Learning with Deep Neural Networks: A Survey
• Deep Learning Based Text Classification: A Comprehensive Review
• Evaluation of Text Generation: A Survey
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• WT5?! Training Text-to-Text Models to Explain their Predictions
• A Survey on Text Classification: From Shallow to Deep Learning
• StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understanding
• How to Fine-Tune BERT for Text Classification?
• MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
• A Surprisingly Robust Trick for Winograd Schema Challenge
• Measuring the Algorithmic Efficiency of Neural Networks
• ERNIE 2.0: A Continual Pre-training Framework for Language Understanding
• ProtTrans: Towards Cracking the Language of Life's Code Through Self-Supervised Deep Learning and High Performance Computing
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• Just Pick a Sign: Optimizing Deep Multitask Models with Gradient Sign Dropout
• Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
• Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inference
• X-SQL: reinforce schema representation with context
• Stochastic Answer Networks for Natural Language Inference
• Provable Meta-Learning of Linear Representations
• 12-in-1: Multi-Task Vision and Language Representation Learning
• MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers
• MixKD: Towards Efficient Distillation of Large-scale Language Models
• TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrieval
• Unifying Question Answering, Text Classification, and Regression via Span Extraction
• TRANS-BLSTM: Transformer with Bidirectional LSTM for Language Understanding
• Reducing Sentiment Bias in Language Models via Counterfactual Evaluation
• BERT Learns (and Teaches) Chemistry
• TreeBERT: A Tree-Based Pre-Trained Model for Programming Language
• CORD19STS: COVID-19 Semantic Textual Similarity Dataset
• Towards non-toxic landscapes: Automatic toxic comment detection using DNN
• MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models
• Multitask Balanced and Recalibrated Network for Medical Code Prediction
• To Tune or Not To Tune? How About the Best of Both Worlds?
• Towards Domain Adaptation from Limited Data for Question Answering Using Deep Neural Networks
• Multi-Task Bidirectional Transformer Representations for Irony Detection
• Meta-Learning with Graph Neural Networks: Methods and Applications
• MMM: Multi-stage Multi-task Learning for Multi-choice Reading Comprehension
• Improve Transformer Models with Better Relative Position Embeddings
• Story Ending Prediction by Transferable BERT
• Generalization in multitask deep neural classifiers: a statistical physics approach
• VIOLIN: A Large-Scale Dataset for Video-and-Language Inference
• Pre-training Text Representations as Meta Learning
• A Hybrid Neural Network Model for Commonsense Reasoning
• Structuring Latent Spaces for Stylized Response Generation
• Multi-task Learning with Sample Re-weighting for Machine Reading Comprehension
• Self-Supervised Dialogue Learning
• Near-optimal Representation Learning for Linear Bandits and Linear RL
• HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections
• Multi-Task Learning for Situated Multi-Domain End-to-End Dialogue Systems
• FLEX: Unifying Evaluation for Few-Shot NLP
• When Choosing Plausible Alternatives, Clever Hans can be Clever
• Domain-Relevant Embeddings for Medical Question Similarity
• Path-Based Contextualization of Knowledge Graphs for Textual Entailment
• Improving Generalization by Incorporating Coverage in Natural Language Inference
• How Does Adversarial Fine-Tuning Benefit BERT?
• Investigating Meta-Learning Algorithms for Low-Resource Natural Language Understanding Tasks
• Ladder Loss for Coherent Visual-Semantic Embedding
• Improving Aspect-Level Sentiment Analysis with Aspect Extraction
• An Empirical Study of Finding Similar Exercises
• Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining
• Enriching Conversation Context in Retrieval-based Chatbots
• Unsupervised Domain Adaptation on Reading Comprehension
• Robustly Optimized and Distilled Training for Natural Language Understanding
• Probing the Natural Language Inference Task with Automated Reasoning Tools
• WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia
• Probing Contextualized Sentence Representations with Visual Awareness
• Learning Latent Parameters without Human Response Patterns: Item Response Theory with Artificial Crowds
• Hierarchical Multi Task Learning with Subword Contextual Embeddings for Languages with Rich Morphology
• ANA at SemEval-2020 Task 4: mUlti-task learNIng for cOmmonsense reasoNing (UNION)
• Multiplicative Position-aware Transformer Models for Language Understanding
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Pentagon at MEDIQA 2019: Multi-task Learning for Filtering and Re-ranking Answers using Language Inference and Question Entailment
• Introducing Pose Consistency and Warp-Alignment for Self-Supervised 6D Object Pose Estimation in Color Images
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference
• Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing
• Memeify: A Large-Scale Meme Generation System
• On the relationship between multitask neural networks and multitask Gaussian Processes
• Shareable Representations for Search Query Understanding
• Pretrained Transformers for Simple Question Answering over Knowledge Graphs
• An Evaluation of Transfer Learning for Classifying Sales Engagement Emails at Large Scale
• A Morpho-Syntactically Informed LSTM-CRF Model for Named Entity Recognition
• Surf at MEDIQA 2019: Improving Performance of Natural Language Inference in the Clinical Domain by Adopting Pre-trained Language Model
• Semi-supervised Visual Feature Integration for Pre-trained Language Models
• R${}^2$-Net: Relation of Relation Learning Network for Sentence Semantic Matching
• MTLHealth: A Deep Learning System for Detecting Disturbing Content in Student Essays