Knowledge Acquisition During Pre-training? Large Language Models Learn Better With Auxiliary Views

Joseph Lee, Yidi Huang, Dokyoon Kim +2

Gaps remain in our understanding of how large language models (LLMs) acquire knowledge during pre-training. We posit that auxiliary views, reformulations of knowledge, are causally…