Looped GPT-BERT: Trading Parameters for Computation in Small Language Modeling

Tingshuo Fan, Hongtao Mu, Tianyu Zhou +2

When training data are limited, increasing parameter count is not the only way to improve language-model performance. A small parameter set, when repeatedly applied, can also deliv…