MOM: Memory-Efficient Offloaded Mini-Sequence Inference for Long Context Language Models

Junyang Zhang, Tianyi Zhu, Cheng Luo +1

Long-context language models exhibit impressive performance but remain challenging to deploy due to high GPU memory demands during inference. We propose Memory-efficient Offloaded…