KoCo: Conditioning Language Model Pre-training on Knowledge Coordinates

Yudong Li, Jiawei Cai, Linlin Shen

Standard Large Language Model (LLM) pre-training typically treats corpora as flattened token sequences, often overlooking the real-world context that humans naturally rely on to co…