Giraffe: Adventures in Expanding Context Lengths in LLMs

Arka Pal, Deep Karkhanis, Manley Roberts +3

Modern large language models (LLMs) that rely on attention mechanisms are typically trained with fixed context lengths which enforce upper limits on the length of input sequences t…