Context-level Language Modeling by Learning Predictive Context Embeddings

Beiya Dai, Yuliang Liu, Daozheng Xue +6

We propose ContextLM, a framework that implicitly learns multi-token prediction by augmenting standard pretraining with an intrinsic next-context prediction objective. ContextLM bu…