TokenRing: An Efficient Parallelism Framework for Infinite-Context LLMs via Bidirectional Communication

Zongwu Wang, Fangxin Liu, Mingshuai Li +1

Efficient parallelization of Large Language Models (LLMs) with long sequences is essential but challenging due to their significant computational and memory demands, particularly s…