Writing in the Margins: Better Inference Pattern for Long Context Retrieval

Melisa Russak, Umar Jamil, Christopher Bryant +4

In this paper, we introduce Writing in the Margins (WiM), a new inference pattern for Large Language Models designed to optimize the handling of long input sequences in retrieval-o…