LongSkywork: A Training Recipe for Efficiently Extending Context Length in Large Language Models

Liang Zhao, Tianwen Wei, Liang Zeng +12

We introduce LongSkywork, a long-context Large Language Model (LLM) capable of processing up to 200,000 tokens. We provide a training recipe for efficiently extending context lengt…