The Low-Dimensional Linear Geometry of Contextualized Word Representations

Evan Hernandez, Jacob Andreas

Black-box probing models can reliably extract linguistic features like tense, number, and syntactic role from pretrained word representations. However, the manner in which these fe…