On the Limits of Minimal Pairs in Contrastive Evaluation

Jannis Vamvas, Rico Sennrich

Minimal sentence pairs are frequently used to analyze the behavior of language models. It is often assumed that model behavior on contrastive pairs is predictive of model behavior…