984 citations · 3.2k across the 49 of their papers we have counts for
13 papers · 1 filter
BLiMP: The Benchmark of Linguistic Minimal Pairs for English
Alex Warstadt, Alicia Parrish, Haokun Liu +4
We introduce The Benchmark of Linguistic Minimal Pairs (shortened to BLiMP), a challenge set for evaluating what language models (LMs) know about major grammatical phenomena in Eng…
Do Attention Heads in BERT Track Syntactic Dependencies?
Phu Mon Htut, Jason Phang, Shikha Bordia +1
We investigate the extent to which individual attention heads in pretrained transformer language models, such as BERT and RoBERTa, implicitly capture syntactic dependency relations…
Inducing Constituency Trees through Neural Machine Translation
Phu Mon Htut, Kyunghyun Cho, Samuel R. Bowman
Latent tree learning(LTL) methods learn to parse sentences using only indirect supervision from a downstream task. Recent advances in latent tree learning have made it possible to…
Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIs
Alex Warstadt, Yu Cao, Ioana Grosu +13
Though state-of-the-art sentence representation models can perform tasks requiring significant knowledge of grammar, it is an open question how best to evaluate their grammatical k…
Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Set
Katharina Kann, Kyunghyun Cho, Samuel R. Bowman
Development sets are impractical to obtain for real low-resource languages, since using all available data for training is often more effective. However, development sets are widel…
Can Unconditional Language Models Recover Arbitrary Sentences?
Nishant Subramani, Samuel R. Bowman, Kyunghyun Cho
Neural network-based generative language models like ELMo and BERT can work effectively as general purpose sentence encoders in text classification without further fine-tuning. Is…
Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark
Nikita Nangia, Samuel R. Bowman
The GLUE benchmark (Wang et al., 2019b) is a suite of language understanding tasks which has seen dramatic progress in the past year, with average performance moving from 70.0 at l…
What do you learn from context? Probing for sentence structure in contextualized word representations
Ian Tenney, Patrick Xia, Berlin Chen +8
Contextualized representation models such as ELMo (Peters et al., 2018a) and BERT (Devlin et al., 2018) have recently achieved state-of-the-art results on a diverse array of downst…
SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems
Alex Wang, Yada Pruksachatkun, Nikita Nangia +5
In the last year, new models and methods for pretraining and transfer learning have driven striking performance improvements across a range of language understanding tasks. The GLU…
Identifying and Reducing Gender Bias in Word-Level Language Models
Shikha Bordia, Samuel R. Bowman
Many text corpora exhibit socially problematic biases, which can be propagated or amplified in the models trained on such data. For example, doctor cooccurs more frequently with ma…
Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Najoung Kim, Roma Patel, Adam Poliak +9
We introduce a set of nine challenge tasks that test for the understanding of function words. These tasks are created by structurally mutating sentences from existing datasets to t…
On Measuring Social Biases in Sentence Encoders
Chandler May, Alex Wang, Shikha Bordia +2
The Word Embedding Association Test shows that GloVe and word2vec word embeddings exhibit human-like implicit biases based on gender, race, and other social constructs (Caliskan et…
Linguistic Analysis of Pretrained Sentence Encoders with Acceptability Judgments
Alex Warstadt, Samuel R. Bowman
Recent work on evaluating grammatical knowledge in pretrained sentence encoders gives a fine-grained view of a small number of phenomena. We introduce a new analysis dataset that a…