GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding

Abstract

For natural language understanding (NLU) technology to be maximally useful, both practically and as a scientific object of study, it must be general: it must be able to process language in a way that is not exclusively tailored to any one specific task or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation benchmark (GLUE), a tool for evaluating and analyzing the performance of models across a diverse range of existing NLU tasks. GLUE is model-agnostic, but it incentivizes sharing knowledge across tasks because certain tasks have very limited training data. We further provide a hand-crafted diagnostic test suite that enables detailed linguistic analysis of NLU models. We evaluate baselines based on current methods for multi-task and transfer learning and find that they do not immediately give substantial improvements over the aggregate performance of training a separate model per task, indicating room for improvement in developing general and robust NLU systems.

arXiv commentICLR 2019; https://gluebenchmark.com/

References in corpus (13)

• Natural Language Processing (almost) from Scratch
• Distributed Representations of Sentences and Documents
• Bidirectional Attention Flow for Machine Comprehension
• Skip-Thought Vectors
• Reasoning about Entailment with Neural Attention
• Dynamic Coattention Networks For Question Answering
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books
• Natural Language Inference over Interaction Space
• Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment
• Transforming Question Answering Datasets Into Natural Language Inference Datasets
• DisSent: Sentence Representation Learning from Explicit Discourse Relations
• Recurrent Neural Network-Based Sentence Encoder with Gated Attention for Natural Language Inference

Cited by in corpus (115)

• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extraction
• Distilling Task-Specific Knowledge from BERT into Simple Neural Networks
• Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges
• Big Bird: Transformers for Longer Sequences
• P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• CLEAR: Contrastive Learning for Sentence Representation
• CvT: Introducing Convolutions to Vision Transformers
• Learning and Evaluating General Linguistic Intelligence
• Parameter-Efficient Transfer Learning for NLP
• Abstraction and Analogy-Making in Artificial Intelligence
• Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks
• Arabic Offensive Language on Twitter: Analysis and Experiments
• Multi-task learning for natural language processing in the 2020s: where are we going?
• On Identifiability in Transformers
• Contrastive Out-of-Distribution Detection for Pretrained Transformers
• Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT
• Reweighted Proximal Pruning for Large-Scale Language Representation
• Natural Language Understanding with Privacy-Preserving BERT
• Using Prior Knowledge to Guide BERT's Attention in Semantic Textual Matching Tasks
• Practical Text Classification With Large Pre-Trained Language Models
• Container: Context Aggregation Network
• LogME: Practical Assessment of Pre-trained Models for Transfer Learning
• EmotionX-IDEA: Emotion BERT -- an Affectional Model for Conversation
• Syntax-Infused Transformer and BERT models for Machine Translation and Natural Language Understanding
• IndoLEM and IndoBERT: A Benchmark Dataset and Pre-trained Language Model for Indonesian NLP
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping
• Are we pretraining it right? Digging deeper into visio-linguistic pretraining
• Transfer Fine-Tuning: A BERT Case Study
• DDPNAS: Efficient Neural Architecture Search via Dynamic Distribution Pruning
• PECOS: Prediction for Enormous and Correlated Output Spaces
• NormFormer: Improved Transformer Pretraining with Extra Normalization
• How fine can fine-tuning be? Learning efficient language models
• Video Understanding as Machine Translation
• A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks
• Length-controllable Abstractive Summarization by Guiding with Summary Prototype
• Testing the limits of natural language models for predicting human language judgments
• Representation learning for neural population activity with Neural Data Transformers
• Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
• Knowledge Distillation from Internal Representations
• A Comparative Study of Transformer-Based Language Models on Extractive Question Answering
• Overview of the CLEF--2021 CheckThat! Lab on Detecting Check-Worthy Claims, Previously Fact-Checked Claims, and Fake News
• Beyond Leaderboards: A survey of methods for revealing weaknesses in Natural Language Inference data and models
• SBERT-WK: A Sentence Embedding Method by Dissecting BERT-based Word Models
• LMSanitator: Defending Prompt-Tuning Against Task-Agnostic Backdoors
• On the comparability of Pre-trained Language Models
• Ranking and Tuning Pre-trained Models: A New Paradigm for Exploiting Model Hubs
• Universal Adversarial Perturbation for Text Classification
• Transfer Learning from Transformers to Fake News Challenge Stance Detection (FNC-1) Task
• MC-BERT: Efficient Language Pre-Training via a Meta Controller
• On Learning Universal Representations Across Languages
• Self-Guided Contrastive Learning for BERT Sentence Representations
• Evolving Attention with Residual Convolutions
• Gradient-Based Adversarial Training on Transformer Networks for Detecting Check-Worthy Factual Claims
• Variance-reduced Language Pretraining via a Mask Proposal Network
• Revisiting BFloat16 Training
• ALP-KD: Attention-Based Layer Projection for Knowledge Distillation
• BERT-EMD: Many-to-Many Layer Mapping for BERT Compression with Earth Mover's Distance
• Generation-Distillation for Efficient Natural Language Understanding in Low-Data Settings
• Code to Comment "Translation": Data, Metrics, Baselining & Evaluation
• Task Selection Policies for Multitask Learning
• ColdGANs: Taming Language GANs with Cautious Sampling Strategies
• LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding
• verBERT: Automating Brazilian Case Law Document Multi-label Categorization Using BERT
• Pay Attention when Required
• Sometimes We Want Translationese
• Modelling Latent Skills for Multitask Language Generation
• LinCE: A Centralized Benchmark for Linguistic Code-switching Evaluation
• Preserving Semantics in Textual Adversarial Attacks
• Undivided Attention: Are Intermediate Layers Necessary for BERT?
• BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
• A semantic embedding space based on large language models for modelling human beliefs
• BloomNet: A Robust Transformer based model for Bloom's Learning Outcome Classification
• Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
• On the attribution of confidence to large language models
• TAG: Gradient Attack on Transformer-based Language Models
• doc2dial: A Goal-Oriented Document-Grounded Dialogue Dataset
• What Meaning-Form Correlation Has to Compose With
• Bilingual Language Modeling, A transfer learning technique for Roman Urdu
• NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance
• WikiCheck: An end-to-end open source Automatic Fact-Checking API based on Wikipedia
• Combining Axiom Injection and Knowledge Base Completion for Efficient Natural Language Inference
• EasyTransfer -- A Simple and Scalable Deep Transfer Learning Platform for NLP Applications
• The Role of Evolution in Machine Intelligence
• Follow Your Path: a Progressive Method for Knowledge Distillation
• Quantifying the Task-Specific Information in Text-Based Classifications
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity
• R&R: Metric-guided Adversarial Sentence Generation
• A Simple Recurrent Unit with Reduced Tensor Product Representations
• Generating Challenge Datasets for Task-Oriented Conversational Agents through Self-Play
• Exceeding the Limits of Visual-Linguistic Multi-Task Learning
• Reliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark
• Lexicosyntactic Inference in Neural Models
• Probing Pre-Trained Language Models for Disease Knowledge
• Empirical Evaluation of Multi-task Learning in Deep Neural Networks for Natural Language Processing
• Training Dynamic based data filtering may not work for NLP datasets
• Neural Architecture Search for Sentence Classification with BERT
• Meta-learning for downstream aware and agnostic pretraining
• Towards A Measure Of General Machine Intelligence
• Indian Legal NLP Benchmarks : A Survey
• Why Can You Lay Off Heads? Investigating How BERT Heads Transfer
• Noise Stability Regularization for Improving BERT Fine-tuning
• Data-Informed Global Sparseness in Attention Mechanisms for Deep Neural Networks
• Predictions For Pre-training Language Models
• Binaural Audio Generation via Multi-task Learning
• Predicting Themes within Complex Unstructured Texts: A Case Study on Safeguarding Reports
• Do Large Language Models know who did what to whom?
• Small-Bench NLP: Benchmark for small single GPU trained models in Natural Language Processing
• A Web Scale Entity Extraction System
• Modeling Performance in Open-Domain Dialogue with PARADISE
• Multi-Granularity Representations of Dialog
• Effective Unsupervised Domain Adaptation with Adversarially Trained Language Models
• Long Short-Term Sample Distillation