Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset

Simon P. Ramalepe, Thipe I. Modipa, Marelie H. Davel

Due to the scarcity of data in low-resourced languages, the development of language models for these languages has been very slow. Currently, pre-trained language models have gaine…