Predicting Emergent Capabilities by Finetuning

Charlie Snell, Eric Wallace, Dan Klein +1

A fundamental open challenge in modern LLM scaling is the lack of understanding around emergent capabilities. In particular, language model pretraining loss is known to be highly p…