Short Data, Long Context: Distilling Positional Knowledge in Transformers

Patrick Huber, Ernie Chang, Chinnadhurai Sankar +4

Extending the context window of language models typically requires expensive long-context pre-training, posing significant challenges for both training efficiency and data collecti…