BERMo: What can BERT learn from ELMo?

Sangamesh Kodge, Kaushik Roy

We propose BERMo, an architectural modification to BERT, which makes predictions based on a hierarchy of surface, syntactic and semantic language features. We use linear combinatio…