IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation

Tianyi Zhang, Florian Mai, Lucie Flek

Continual pretraining promises to adapt large language models (LLMs) to new domains using only unlabeled test-time data, but naively applying standard self-supervised objectives to…