The Impossibility Triangle of Long-Context Modeling

Yan Zhou

We identify and prove a fundamental trade-off governing long-sequence models: no model can simultaneously achieve (i) per-step computation independent of sequence length (Efficienc…