SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems

Abstract

In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLUE benchmark, introduced a little over one year ago, offers a single-number metric that summarizes progress on a diverse set of such tasks, but performance on the benchmark has recently surpassed the level of non-expert humans, suggesting limited headroom for further research. In this paper we present SuperGLUE, a new benchmark styled after GLUE with a new set of more difficult language understanding tasks, a software toolkit, and a public leaderboard. SuperGLUE is available at super.gluebenchmark.com.

NeurIPS 2019, super.gluebenchmark.com updating acknowledegments

References in corpus (10)

• Distilling the Knowledge in a Neural Network
• Skip-Thought Vectors
• Semi-supervised Sequence Learning
• The Natural Language Decathlon: Multitask Learning as Question Answering
• Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Tasks
• ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• What do you learn from context? Probing for sentence structure in contextualized word representations
• A Surprisingly Robust Trick for Winograd Schema Challenge

Cited by in corpus (84)

• Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing
• Pre-trained Models for Natural Language Processing: A Survey
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• Self-Supervised Representation Learning: Introduction, Advances and Challenges
• The Debate Over Understanding in AI's Large Language Models
• TabFact: A Large-scale Dataset for Table-based Fact Verification
• The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures
• Factuality Challenges in the Era of Large Language Models
• A Survey on Aspect-Based Sentiment Classification
• Advances of Machine Learning in Materials Science: Ideas and Techniques
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Multi-task learning for natural language processing in the 2020s: where are we going?
• Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora
• Improving Short Text Classification With Augmented Data Using GPT-3
• Adversarial NLI: A New Benchmark for Natural Language Understanding
• FlauBERT: Unsupervised Language Model Pre-training for French
• SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training
• Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding
• LEXTREME: A Multi-Lingual and Multi-Task Benchmark for the Legal Domain
• Towards Zero-Label Language Learning
• Grad-SAM: Explaining Transformers via Gradient Self-Attention Maps
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing
• FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement
• Do LLMs Understand Social Knowledge? Evaluating the Sociability of Large Language Models with SocKET Benchmark
• Advancing Neural Encoding of Portuguese with Transformer Albertina PT-*
• Several categories of Large Language Models (LLMs): A Short Survey
• Clinical Insights: A Comprehensive Review of Language Models in Medicine
• Two Directions for Clinical Data Generation with Large Language Models: Data-to-Label and Label-to-Data
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Testing the limits of natural language models for predicting human language judgments
• Introducing MBIB -- the first Media Bias Identification Benchmark Task and Dataset Collection
• A New Dataset for Natural Language Inference from Code-mixed Conversations
• FCGEC: Fine-Grained Corpus for Chinese Grammatical Error Correction
• Natural Language Processing with Commonsense Knowledge: A Survey
• GLU Variants Improve Transformer
• The Effect of Third Party Implementations on Reproducibility
• Testing the Generalization of Neural Language Models for COVID-19 Misinformation Detection
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization
• ORCAS-I: Queries Annotated with Intent using Weak Supervision
• KorNLI and KorSTS: New Benchmark Datasets for Korean Natural Language Understanding
• HULK: An Energy Efficiency Benchmark Platform for Responsible Natural Language Processing
• Structured Reordering for Modeling Latent Alignments in Sequence Transduction
• Lexical semantics enhanced neural word embeddings
• Domain-Specific Pretraining for Vertical Search: Case Study on Biomedical Literature
• Improving Information Extraction on Business Documents with Specific Pre-Training Tasks
• Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models
• DaNetQA: a yes/no Question Answering Dataset for the Russian Language
• Ensemble Transformer for Efficient and Accurate Ranking Tasks: an Application to Question Answering Systems
• Probing Neural Dialog Models for Conversational Understanding
• Analysis of the Evolution of Advanced Transformer-Based Language Models: Experiments on Opinion Mining
• The Dark Side of the Language: Pre-trained Transformers in the DarkNet
• LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization
• ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks
• What Context Features Can Transformer Language Models Use?
• SMILE: Evaluation and Domain Adaptation for Social Media Language Understanding
• SuperSim: a test set for word similarity and relatedness in Swedish
• Scalable Multitask Learning Using Gradient-based Estimation of Task Affinity
• Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
• TAPE: Assessing Few-shot Russian Language Understanding
• When Do Discourse Markers Affect Computational Sentence Understanding?
• Vote'n'Rank: Revision of Benchmarking with Social Choice Theory
• Beyond Distributional Hypothesis: Let Language Models Learn Meaning-Text Correspondence
• BEAMetrics: A Benchmark for Language Generation Evaluation Evaluation
• Shatter: An Efficient Transformer Encoder with Single-Headed Self-Attention and Relative Sequence Partitioning
• X-PuDu at SemEval-2022 Task 7: A Replaced Token Detection Task Pre-trained Model with Pattern-aware Ensembling for Identifying Plausible Clarifications
• Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarks
• Modelling Language using Large Language Models
• Analyzing Multi-Task Learning for Abstractive Text Summarization
• Evaluating the Effects of Prompt Perturbation on Bias and Hallucination in Large Language Models
• Reliable Evaluations for Natural Language Inference based on a Unified Cross-dataset Benchmark
• Toward a benchmark for CTR prediction in online advertising: datasets, evaluation protocols and perspectives
• CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
• Explaining Relation Classification Models with Semantic Extents
• Correcting diacritics and typos with a ByT5 transformer model
• Towards Tractable Mathematical Reasoning: Challenges, Strategies, and Opportunities for Solving Math Word Problems
• MetaXT: Meta Cross-Task Transfer between Disparate Label Spaces
• Review on 6D Object Pose Estimation with the focus on Indoor Scene Understanding
• Increasing The Performance of Cognitively Inspired Data-Efficient Language Models via Implicit Structure Building
• Exploring a Unified Sequence-To-Sequence Transformer for Medical Product Safety Monitoring in Social Media
• Calibrating Structured Output Predictors for Natural Language Processing
• Composition, Attention, or Both?
• Injecting Hierarchy with U-Net Transformers
• Unsupervised Pronoun Resolution via Masked Noun-Phrase Prediction
• Cheap Learning: Maximising Performance of Language Models for Social Data Science Using Minimal Data