Diffract: Spectral View of LLM Domain Adaptation

Nikita Borodin, Maria Krylova, Artem Zabolotnyi +6

We study continual pre-training (CPT) as a mechanism for adapting general-purpose large language models to specialized domains: mathematics, instruction, code, and natural text. Us…