OctoLong: Mid-Training On Cross-Repository Code Contexts Enhances Long-Context Modeling

Indraneil Paul, Falko Helm, Goran Glavaš +1

Context lengths of language models (LMs) have dramatically increased, driven by the demands for in-context learning, self-improvement, and long-horizon agentic workflows. However,…