A Controlled Study on Long Context Extension and Generalization in LLMs

Yi Lu, Jing Nathan Yan, Songlin Yang +6

Broad textual understanding and in-context learning require language models that utilize full document contexts. Due to the implementation challenges associated with directly train…