Arctic Long Sequence Training: Scalable And Efficient Training For Multi-Million Token Sequences

Stas Bekman, Samyam Rajbhandari, Michael Wyatt +5

Long sequences are critical for applications like RAG, long document summarization, multi-modality, etc., and modern LLMs, like Llama 4 Scout, support max sequence length of up to…