HelaBERT: Enhancing Sinhala Language Understanding with Dual Pooling Classification Head

Thisen Ekanayake, Nisansa de Silva

We present HelaBERT, a family of two BERT-based masked language models pre-trained from scratch on approximately 1 billion tokens of Sinhala text sourced from MADLAD-400, CulturaX,…