Not-Just-Scaling Laws: Towards a Better Understanding of the Downstream Impact of Language Model Design Decisions

Emmy Liu, Amanda Bertsch, Lintang Sutawika +9

Improvements in language model capabilities are often attributed to increasing model size or training data, but in some cases smaller models trained on curated data or with differe…