Context Language Models

Rulin Shao, Shannon Zejiang Shen, Junjie Oscar Yin +10

We introduce Context Language Models (CLMs), language models that natively manage their own context. We implement this by treating the context as a file and allowing the model to m…