305 citations · 320 across the 24 of their papers we have counts for
14 papers · 1 filter
LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training
Minqiu Sun, Xin Huang, Luanzheng Guo +3
With the rising computational and monetary costs of training large language models (LLMs), checkpointing---periodically storing model states for recovery---becomes essential for fa…
ZOCheck: CPU-Shadow Checkpointing for Zeroth-Order LLM Fine-Tuning
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress…
Effects of Hybrid CPU and Cache Architectures on Parallel HPC and Cloud Applications
Nanda Velugoti, Joseph Manzano, Nathan Tallent +1
Hybrid CPU architectures have entered the mainstream desktop computing with the announcement of Intel's Alderlake architecture. Such a transition to heterogeneous CPU architecture…
MEMPOWER: Efficient Power Management with Fine-grained Memory Analysis and Modeling for HPC Workloads
Nanda Velugoti, Joseph Manzano, Andres Marquez +2
Managing the energy consumption and power efficiency of parallel applications is a significant issue in both HPC environments and in the cloud. As emerging applications continue to…
QoSFlow: Ensuring Service Quality of Distributed Workflows Using Interpretable Sensitivity Models
Md Hasanur Rashid, Jesun Firoz, Nathan R. Tallent +3
With the increasing importance of distributed scientific workflows, there is a critical need to ensure Quality of Service (QoS) constraints, such as minimizing time or limiting exe…
CARAT: Client-Side Adaptive RPC and Cache Co-Tuning for Parallel File Systems
Md Hasanur Rashid, Nathan R. Tallent, Forrest Sheng Bao +1
Tuning parallel file system in High-Performance Computing (HPC) systems remains challenging due to the complex I/O paths, diverse I/O patterns, and dynamic system conditions. While…
LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Checkpointing is essential for fault tolerance in training large language models (LLMs). However, existing methods, regardless of their I/O strategies, periodically store the entir…
On The Reproducibility Limitations of RAG Systems
Baiqiang Wang, Dongfang Zhao, Nathan R Tallent +1
Retrieval-Augmented Generation (RAG) is increasingly employed in generative AI-driven scientific workflows to integrate rapidly evolving scientific knowledge bases, yet its reliabi…
PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training
Talha Mehboob, Luanzheng Guo, Nathan Tallent +2
The exponential growth of large-scale AI models has led to computational and power demands that can exceed the capacity of a single data center. This is due to the limited power su…
MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs
Aishwarya Sarkar, Sayan Ghosh, Nathan R. Tallent +1
Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significa…
Final Report for CHESS: Cloud, High-Performance Computing, and Edge for Science and Security
Nathan Tallent, Jan Strube, Luanzheng Guo +9
Automating the theory-experiment cycle requires effective distributed workflows that utilize a computing continuum spanning lab instruments, edge sensors, computing resources at mu…
Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows
Rafael Ferreira da Silva, Deborah Bard, Kyle Chard +108
The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensit…
Overcoming Memory Constraints in Quantum Circuit Simulation with a High-Fidelity Compression Framework
Boyuan Zhang, Bo Fang, Fanjiang Ye +4
Full-state quantum circuit simulation requires exponentially increased memory size to store the state vector as the number of qubits scales, presenting significant limitations in c…
Bit-GraphBLAS: Bit-Level Optimizations of Matrix-Centric Graph Processing on GPU
Jou-An Chen, Hsin-Hsuan Sung, Xipeng Shen +3
In a general graph data structure like an adjacency matrix, when edges are homogeneous, the connectivity of two nodes can be sufficiently represented using a single bit. This insig…