Spying on your neighbors: Fine-grained probing of contextual embeddings for information about surrounding words

Josef Klafka, Allyson Ettinger

Although models using contextual word embeddings have achieved state-of-the-art results on a host of NLP tasks, little is known about exactly what information these embeddings enco…