Detecting and Exorcising Statistical Demons from Language Models with Anti-Models of Negative Data

Michael L. Wick, Kate Silverstein, Jean-Baptiste Tristan +2

It's been said that "Language Models are Unsupervised Multitask Learners." Indeed, self-supervised language models trained on "positive" examples of English text generalize in desi…