LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptation

Xuan Zhang, Fengzhuo Zhang, Cunxiao Du +4

Scaling language models to handle longer contexts introduces substantial memory challenges due to the growing cost of key-value (KV) caches. Motivated by the efficiency gains of hy…