Mergeable Model-Side Aggregation States for Long-Context Language Models

Dachuan Song, Junyu Yin, Zechen Hu +1

A known limitation of long-context language models is their increasingly unreliable performance in non-additive, set-based aggregation as context length grows. Examples include car…