Open Sesame: Getting Inside BERT's Linguistic Knowledge

Yongjie Lin, Yi Chern Tan, Robert Frank

How and to what extent does BERT encode syntactically-sensitive hierarchical information or positionally-sensitive linear information? Recent work has shown that contextual represe…