148 citations · 272 across the 17 of their papers we have counts for
26 papers
Weisfeiler and Leman Go Measurement Modeling: Probing the Validity of the WL Test
Arjun Subramonian, Adina Williams, Maximilian Nickel +2
The expressive power of graph neural networks is usually measured by comparing how many pairs of graphs or nodes an architecture can possibly distinguish as non-isomorphic to those…
Language model acceptability judgements are not always robust to context
Koustuv Sinha, Jon Gauthier, Aaron Mueller +4
Targeted syntactic evaluations of language models ask whether models show stable preferences for syntactically acceptable content over minimal-pair unacceptable inputs. Most target…
The Curious Case of Absolute Position Embeddings
Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy +3
Transformer language models encode the notion of word order using positional information. Most commonly, this positional information is represented by absolute position embeddings…
Benchmarking Compositionality with Formal Languages
Josef Valvoda, Naomi Saphra, Jonathan Rawski +2
Recombining known primitive concepts into larger novel combinations is a quintessentially human cognitive capability. Whether large neural models in NLP can acquire this ability wh…
A Latent-Variable Model for Intrinsic Probing
Karolina Stańczak, Lucas Torroba Hennigen, Adina Williams +2
The success of pre-trained contextualized representations has prompted researchers to analyze them for the presence of linguistic information. Indeed, it is natural to assume that…
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models
Eric Michael Smith, Adina Williams
All AI models are susceptible to learning biases in data that they are trained on. For generative dialogue models, being trained on real human conversations containing unbalanced g…
UnNatural Language Inference
Koustuv Sinha, Prasanna Parthasarathi, Joelle Pineau +1
Recent investigations into the inner-workings of state-of-the-art large-scale pre-trained Transformer-based Natural Language Understanding (NLU) models indicate that they appear to…
To what extent do human explanations of model behavior align with actual model behavior?
Grusha Prasad, Yixin Nie, Mohit Bansal +3
Given the increasingly prominent role NLP models (will) play in our lives, it is important for human expectations of model behavior to align with actual model behavior. Using Natur…
ANLIzing the Adversarial Natural Language Inference Dataset
Adina Williams, Tristan Thrush, Douwe Kiela
We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over mul…
Intrinsic Probing through Dimension Selection
Lucas Torroba Hennigen, Adina Williams, Ryan Cotterell
Most modern NLP systems make use of pre-trained contextual representations that attain astonishingly high performance on a variety of tasks. Such high performance should not be pos…
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection
Ekaterina Vylomova, Jennifer White, Elizabeth Salesky +25
A broad goal in natural language processing (NLP) is to develop a system that has the capacity to process any natural language. Most systems, however, are developed using data from…
Predicting Declension Class from Form and Meaning
Adina Williams, Tiago Pimentel, Arya D. McCarthy +3
The noun lexica of many natural languages are divided into several declension classes with characteristic morphological properties. Class membership is far from deterministic, but…
Information-Theoretic Probing for Linguistic Structure
Tiago Pimentel, Josef Valvoda, Rowan Hall Maudslay +3
The success of neural networks on a diverse set of NLP tasks has led researchers to question how much these networks actually ``know'' about natural language. Probes are a natural…
A Tale of a Probe and a Parser
Rowan Hall Maudslay, Josef Valvoda, Tiago Pimentel +2
Measuring what linguistic information is encoded in neural models of language has become popular in NLP. Researchers approach this enterprise by training "probes" - supervised mode…
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin +2
We use large-scale corpora in six different gendered languages, along with tools from NLP and information theory, to test whether there is a relationship between the grammatical ge…
Multi-Dimensional Gender Bias Classification
Emily Dinan, Angela Fan, Ledell Wu +3
Machine learning models are trained to find patterns in data. NLP models can inadvertently learn socially undesirable patterns when training on gender biased text. In this work, we…
Are Natural Language Inference Models IMPPRESsive? Learning IMPlicature and PRESupposition
Paloma Jeretic, Alex Warstadt, Suvrat Bhooshan +1
Natural language inference (NLI) is an increasingly important task for natural language understanding, which requires one to infer whether a sentence entails another. However, the…
Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation
Emily Dinan, Angela Fan, Adina Williams +3
Models often easily learn biases present in the training data, and their predictions directly reflect this bias. We analyze gender bias in dialogue data, and examine how this bias…
Quantifying the Semantic Core of Gender Systems
Adina Williams, Ryan Cotterell, Lawrence Wolf-Sonkin +2
Many of the world's languages employ grammatical gender on the lexeme. For example, in Spanish, the word for 'house' (casa) is feminine, whereas the word for 'paper' (papel) is mas…
Adversarial NLI: A New Benchmark for Natural Language Understanding
Yixin Nie, Adina Williams, Emily Dinan +3
We introduce a new large-scale NLI benchmark dataset, collected via an iterative, adversarial human-and-model-in-the-loop procedure. We show that training models on this new datase…
On the Idiosyncrasies of the Mandarin Chinese Classifier System
Shijia Liu, Hongyuan Mei, Adina Williams +1
While idiosyncrasies of the Chinese classifier system have been a richly studied topic among linguists (Adams and Conklin, 1973; Erbaugh, 1986; Lakoff, 1986), not much work has bee…
Verb Argument Structure Alternations in Word and Sentence Embeddings
Katharina Kann, Alex Warstadt, Adina Williams +1
Verbs occur in different syntactic environments, or frames. We investigate whether artificial neural networks encode grammatical distinctions necessary for inferring the idiosyncra…
XNLI: Evaluating Cross-lingual Sentence Representations
Alexis Conneau, Guillaume Lample, Ruty Rinott +4
State-of-the-art natural language processing systems rely on supervision in the form of annotated data to learn competent models. These models are generally trained on data in a si…
Do latent tree learning models identify meaningful structure in sentences?
Adina Williams, Andrew Drozdov, Samuel R. Bowman
Recent work on the problem of latent tree learning has made it possible to train neural networks that learn to both parse a sentence and use the resulting parse to interpret the se…