How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

Guanting Dong, Hongyi Yuan, Keming Lu +7

Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, code generation, and instruction following. These…