Entropy-Based Data Selection for Language Models

Hongming Li, Yang Liu, Chao Huang

Modern language models (LMs) increasingly require two critical resources: computational resources and data resources. Data selection techniques can effectively reduce the amount of…