Acquiring Bidirectionality via Large and Small Language Models

Takumi Goto, Hiroyoshi Nagao, Yuta Koreeda

Using token representation from bidirectional language models (LMs) such as BERT is still a widely used approach for token-classification tasks. Even though there exist much larger…