Beyond Scale: The Diversity Coefficient as a Data Quality Metric for Variability in Natural Language Data

Brando Miranda, Alycia Lee, Sudharsan Sundar +4

Current trends in pre-training Large Language Models (LLMs) primarily focus on the scaling of model and dataset size. While the quality of pre-training data is considered an import…