305 citations · 320 across the 24 of their papers we have counts for
11 papers · 1 filter
LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training
Minqiu Sun, Xin Huang, Luanzheng Guo +3
With the rising computational and monetary costs of training large language models (LLMs), checkpointing---periodically storing model states for recovery---becomes essential for fa…
ZOCheck: CPU-Shadow Checkpointing for Zeroth-Order LLM Fine-Tuning
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress…
Effects of Hybrid CPU and Cache Architectures on Parallel HPC and Cloud Applications
Nanda Velugoti, Joseph Manzano, Nathan Tallent +1
Hybrid CPU architectures have entered the mainstream desktop computing with the announcement of Intel's Alderlake architecture. Such a transition to heterogeneous CPU architecture…
MEMPOWER: Efficient Power Management with Fine-grained Memory Analysis and Modeling for HPC Workloads
Nanda Velugoti, Joseph Manzano, Andres Marquez +2
Managing the energy consumption and power efficiency of parallel applications is a significant issue in both HPC environments and in the cloud. As emerging applications continue to…
NOMAD: Generating Embeddings for Massive Distributed Graphs
Aishwarya Sarkar, Sayan Ghosh, Nathan R. Tallent +1
Successful machine learning on graphs or networks requires embeddings that not only represent nodes and edges as low-dimensional vectors but also preserve the graph structure. Esta…
Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training
Cunyang Wei, Siddharth Singh, Aishwarya Sarkar +7
Graph neural networks (GNNs) are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed tra…
SCOPE: Scalable Training-Free Semantic Coreset Selection for Long-Tailed Federated Learning
Md Anwar Hossen, Nathan R. Tallent, Luanzheng Guo +1
Scientific discovery increasingly relies on federated learning over data from high-resolution instruments, and these data are strongly non-IID and long-tailed. Coreset selection co…
QoSFlow: Ensuring Service Quality of Distributed Workflows Using Interpretable Sensitivity Models
Md Hasanur Rashid, Jesun Firoz, Nathan R. Tallent +3
With the increasing importance of distributed scientific workflows, there is a critical need to ensure Quality of Service (QoS) constraints, such as minimizing time or limiting exe…
Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents
Aishwarya Sarkar, Sayan Ghosh, Nathan Tallent +3
Large-scale Graph Neural Networks (GNNs) are typically trained by sampling a vertex's neighbors to a fixed distance. Because large input graphs are distributed, training requires f…
CARAT: Client-Side Adaptive RPC and Cache Co-Tuning for Parallel File Systems
Md Hasanur Rashid, Nathan R. Tallent, Forrest Sheng Bao +1
Tuning parallel file system in High-Performance Computing (HPC) systems remains challenging due to the complex I/O paths, diverse I/O patterns, and dynamic system conditions. While…
LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Checkpointing is essential for fault tolerance in training large language models (LLMs). However, existing methods, regardless of their I/O strategies, periodically store the entir…