MEMO: Fine-grained Tensor Management For Ultra-long Context LLM Training

Pinxue Zhao, Hailin Zhang, Fangcheng Fu +9

Nowadays, Large Language Models (LLMs) have been trained using extended context lengths to foster more creative applications. However, long context training poses great challenges…