most citedDeep Optimizer States: Towards Scalable Training of Transformer Models Using Interleaved Offloading

9 citations · 17 across the 2 of their papers we have counts for

collaborators
Showing cs.DCShow all

1 paper · 1 filter

cs.DC2024★ 8 cited

Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers

Avinash Maurya, Jie Ye, M. Mustafa Rafique +2

Transformers and LLMs have seen rapid adoption in all domains. Their sizes have exploded to hundreds of billions of parameters and keep increasing. Under these circumstances, the t…