Semantics or spelling? Probing contextual word embeddings with orthographic noise

Jacob A. Matthews, John R. Starr, Marten van Schijndel

Pretrained language model (PLM) hidden states are frequently employed as contextual word embeddings (CWE): high-dimensional representations that encode semantic information given l…