2 papers
cs.CL2024
TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewise
Nan He, Hanyu Lai, Chenyang Zhao +12
Large Language Models (LLMs) exhibit impressive reasoning and data augmentation capabilities in various NLP tasks. However, what about small models? In this work, we propose Teache…
cs.CL2024
SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
Nan He, Weichen Xiong, Hanwen Liu +6
The effectiveness of large language models (LLMs) is often hindered by duplicated data in their extensive pre-training datasets. Current approaches primarily focus on detecting and…