9 citations · 9 across the 5 of their papers we have counts for
7 papers · 1 filter
ReCoVer: Resilient LLM Pre-Training System via Fault-Tolerant Collective and Versatile Workload
Ziyue Liu, Zhengyang Wang, Ruijie Zhang +7
Pre-training large language models on massive GPU clusters has made hardware faults routine rather than rare, driving the need for resilient training systems. Yet existing framewor…
Understanding Inference Scaling for LLMs: Bottlenecks, Trade-offs, and Performance Principles
Moiz Arif, Avinash Maurya, Sudharshan Vazhkudai +1
The transition from standard generative AI to \emph{reasoning-centric architectures}, exemplified by models capable of extensive Chain-of-Thought~(CoT) processing, marks a fundamen…
DataStates-LLM: Scalable Checkpointing for Transformer Models Using Composable State Providers
Avinash Maurya, M. Mustafa Rafique, Franck Cappello +1
The rapid growth of Large Transformer-based models, specifically Large Language Models (LLMs), now scaling to trillions of parameters, has necessitated training across thousands of…
Understanding LLM Checkpoint/Restore I/O Strategies and Patterns
Mikaila J. Gossman, Avinash Maurya, Bogdan Nicolae +1
As LLMs and foundation models scale, checkpoint/restore has become a critical pattern for training and inference. With 3D parallelism (tensor, pipeline, data), checkpointing involv…
MLP-Offload: Multi-Level, Multi-Path Offloading for LLM Pre-training to Break the GPU Memory Wall
Avinash Maurya, M. Mustafa Rafique, Franck Cappello +1
Training LLMs larger than the aggregated memory of multiple GPUs is increasingly necessary due to the faster growth of LLM sizes compared to GPU memory. To this end, multi-tier hos…
Breaking the Memory Wall: A Study of I/O Patterns and GPU Memory Utilization for Hybrid CPU-GPU Offloaded Optimizers
Avinash Maurya, Jie Ye, M. Mustafa Rafique +2
Transformers and LLMs have seen rapid adoption in all domains. Their sizes have exploded to hundreds of billions of parameters and keep increasing. Under these circumstances, the t…