984 citations · 3.2k across the 49 of their papers we have counts for
63 papers
SocioProbe: What, When, and Where Language Models Learn about Sociodemographics
Anne Lauscher, Federico Bianchi, Samuel Bowman +1
Pre-trained language models (PLMs) have outperformed other NLP models on a wide range of tasks. Opting for a more thorough understanding of their capabilities and inner workings, r…
What Artificial Neural Networks Can Tell Us About Human Language Acquisition
Alex Warstadt, Samuel R. Bowman
Rapid progress in machine learning for natural language processing has the potential to transform debates about how humans learn language. However, the learning environments and bi…
SQuALITY: Building a Long-Document Summarization Dataset the Hard Way
Alex Wang, Richard Yuanzhe Pang, Angelica Chen +2
Summarization datasets are often assembled either by scraping naturally occurring public-domain summaries -- which are nearly always in difficult-to-work-with technical domains --…
Instruction Induction: From Few Examples to Natural Language Task Descriptions
Or Honovich, Uri Shaham, Samuel R. Bowman +1
Large language models are able to perform a task by conditioning on a few input-output demonstrations - a paradigm known as in-context learning. We show that language models can ex…
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair
Jason Phang, Angelica Chen, William Huang +1
More capable language models increasingly saturate existing task benchmarks, in some cases outperforming humans. This has left little headroom with which to measure further progres…
Clean or Annotate: How to Spend a Limited Data Collection Budget
Derek Chen, Zhou Yu, Samuel R. Bowman
Crowdsourcing platforms are often used to collect datasets for training machine learning models, despite higher levels of inaccurate labeling compared to expert labeling. There are…
Fine-Tuned Transformers Show Clusters of Similar Representations Across Layers
Jason Phang, Haokun Liu, Samuel R. Bowman
Despite the success of fine-tuning pretrained language encoders like BERT for downstream natural language understanding (NLU) tasks, it is still poorly understood how neural networ…
Does Putting a Linguist in the Loop Improve NLU Data Collection?
Alicia Parrish, William Huang, Omar Agha +7
Many crowdsourced NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. Identifying these issues from early data samples durin…
Efficient transfer learning for NLP with ELECTRA
François Mercier
Clark et al. [2020] claims that the ELECTRA approach is highly efficient in NLP performances relative to computation budget. As such, this reproducibility study focus on this claim…
What Will it Take to Fix Benchmarking in Natural Language Understanding?
Samuel R. Bowman, George E. Dahl
Evaluation for many natural language understanding (NLU) tasks is broken: Unreliable and biased systems score so highly on standard benchmarks that there is little room for researc…
When Do You Need Billions of Words of Pretraining Data?
Yian Zhang, Alex Warstadt, Haau-Sing Li +1
NLP is currently dominated by general-purpose pretrained language models like RoBERTa, which achieve strong performance on NLU tasks through pretraining on billions of words. But w…
Asking Crowdworkers to Write Entailment Examples: The Best of Bad Options
Clara Vania, Ruijie Chen, Samuel R. Bowman
Large-scale natural language inference (NLI) datasets such as SNLI or MNLI have been created by asking crowdworkers to read a premise and write three new hypotheses, one for each p…
Learning Which Features Matter: RoBERTa Acquires a Preference for Linguistic Generalizations (Eventually)
Alex Warstadt, Yian Zhang, Haau-Sing Li +2
One reason pretraining on self-supervised linguistic tasks is effective is that it teaches models features that are helpful for language understanding. However, we want pretrained…
Counterfactually-Augmented SNLI Training Data Does Not Yield Better Generalization Than Unaugmented Data
William Huang, Haokun Liu, Samuel R. Bowman
A growing body of work shows that models exploit annotation artifacts to achieve state-of-the-art performance on standard crowdsourced benchmarks---datasets collected from crowdwor…
Precise Task Formalization Matters in Winograd Schema Evaluations
Haokun Liu, William Huang, Dhara A. Mungra +1
Performance on the Winograd Schema Challenge (WSC), a respected English commonsense reasoning benchmark, recently rocketed from chance accuracy to 89% on the SuperGLUE leaderboard,…
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
Nikita Nangia, Clara Vania, Rasika Bhalerao +1
Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases tha…
Can neural networks acquire a structural bias from raw linguistic data?
Alex Warstadt, Samuel R. Bowman
We evaluate whether BERT, a widely used neural network for sentence processing, acquires an inductive bias towards forming structural generalizations through pretraining on raw dat…
Self-Training for Unsupervised Parsing with PRPN
Anhad Mohananey, Katharina Kann, Samuel R. Bowman
Neural unsupervised parsing (UP) models learn to parse without access to syntactic annotations, while being optimized for another task like language modeling. In this work, we prop…
Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?
Yada Pruksachatkun, Jason Phang, Haokun Liu +6
While pretrained models such as BERT have shown large gains across natural language understanding tasks, their performance can be improved by further training the model on a data-r…
English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too
Jason Phang, Iacer Calixto, Phu Mon Htut +5
Intermediate-task training---fine-tuning a pretrained model on an intermediate task before fine-tuning again on the target task---often improves model performance substantially on…
Learning to Learn Morphological Inflection for Resource-Poor Languages
Katharina Kann, Samuel R. Bowman, Kyunghyun Cho
We propose to cast the task of morphological inflection - mapping a lemma to an indicated inflected form - for resource-poor languages as a meta-learning problem. Treating each lan…
New Protocols and Negative Results for Textual Entailment Data Collection
Samuel R. Bowman, Jennimaria Palomaki, Livio Baldini Soares +1
Natural language inference (NLI) data has proven useful in benchmarking and, especially, as pretraining data for tasks requiring language understanding. However, the crowdsourcing…
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models
Yada Pruksachatkun, Phil Yeres, Haokun Liu +5
We introduce jiant, an open source toolkit for conducting multitask and transfer learning experiments on English NLU tasks. jiant enables modular and configuration-driven experimen…
BLiMP: The Benchmark of Linguistic Minimal Pairs for English
Alex Warstadt, Alicia Parrish, Haokun Liu +4
We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in Eng…