305 citations · 320 across the 24 of their papers we have counts for
24 papers
LayerCheck: Adaptive Layer-wise Checkpointing for Large Language Model Post-training
Minqiu Sun, Xin Huang, Luanzheng Guo +3
With the rising computational and monetary costs of training large language models (LLMs), checkpointing---periodically storing model states for recovery---becomes essential for fa…
ZOCheck: CPU-Shadow Checkpointing for Zeroth-Order LLM Fine-Tuning
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Zeroth-order (ZO) optimization is an attractive option for memory-efficient LLM fine-tuning, but its fault tolerance remains underexplored. Unlike first-order training, ZO progress…
Effects of Hybrid CPU and Cache Architectures on Parallel HPC and Cloud Applications
Nanda Velugoti, Joseph Manzano, Nathan Tallent +1
Hybrid CPU architectures have entered the mainstream desktop computing with the announcement of Intel's Alderlake architecture. Such a transition to heterogeneous CPU architecture…
MEMPOWER: Efficient Power Management with Fine-grained Memory Analysis and Modeling for HPC Workloads
Nanda Velugoti, Joseph Manzano, Andres Marquez +2
Managing the energy consumption and power efficiency of parallel applications is a significant issue in both HPC environments and in the cloud. As emerging applications continue to…
NOMAD: Generating Embeddings for Massive Distributed Graphs
Aishwarya Sarkar, Sayan Ghosh, Nathan R. Tallent +1
Successful machine learning on graphs or networks requires embeddings that not only represent nodes and edges as low-dimensional vectors but also preserve the graph structure. Esta…
Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training
Cunyang Wei, Siddharth Singh, Aishwarya Sarkar +7
Graph neural networks (GNNs) are widely used for learning on graph datasets derived from various real-world scenarios. Learning from extremely large graphs requires distributed tra…
SCOPE: Scalable Training-Free Semantic Coreset Selection for Long-Tailed Federated Learning
Md Anwar Hossen, Nathan R. Tallent, Luanzheng Guo +1
Scientific discovery increasingly relies on federated learning over data from high-resolution instruments, and these data are strongly non-IID and long-tailed. Coreset selection co…
QoSFlow: Ensuring Service Quality of Distributed Workflows Using Interpretable Sensitivity Models
Md Hasanur Rashid, Jesun Firoz, Nathan R. Tallent +3
With the increasing importance of distributed scientific workflows, there is a critical need to ensure Quality of Service (QoS) constraints, such as minimizing time or limiting exe…
Rudder: Steering Prefetching in Distributed GNN Training using LLM Agents
Aishwarya Sarkar, Sayan Ghosh, Nathan Tallent +3
Large-scale Graph Neural Networks (GNNs) are typically trained by sampling a vertex's neighbors to a fixed distance. Because large input graphs are distributed, training requires f…
CARAT: Client-Side Adaptive RPC and Cache Co-Tuning for Parallel File Systems
Md Hasanur Rashid, Nathan R. Tallent, Forrest Sheng Bao +1
Tuning parallel file system in High-Performance Computing (HPC) systems remains challenging due to the complex I/O paths, diverse I/O patterns, and dynamic system conditions. While…
LLMTailor: A Layer-wise Tailoring Tool for Efficient Checkpointing of Large Language Models
Minqiu Sun, Xin Huang, Luanzheng Guo +3
Checkpointing is essential for fault tolerance in training large language models (LLMs). However, existing methods, regardless of their I/O strategies, periodically store the entir…
ProHD: Projection-Based Hausdorff Distance Approximation
Jiuzhou Fu, Luanzheng Guo, Nathan R. Tallent +1
The Hausdorff distance (HD) is a robust measure of set dissimilarity, but computing it exactly on large, high-dimensional datasets is prohibitively expensive. We propose \textbf{Pr…
On The Reproducibility Limitations of RAG Systems
Baiqiang Wang, Dongfang Zhao, Nathan R Tallent +1
Retrieval-Augmented Generation (RAG) is increasingly employed in generative AI-driven scientific workflows to integrate rapidly evolving scientific knowledge bases, yet its reliabi…
PowerTrip: Exploiting Federated Heterogeneous Datacenter Power for Distributed ML Training
Talha Mehboob, Luanzheng Guo, Nathan Tallent +2
The exponential growth of large-scale AI models has led to computational and power demands that can exceed the capacity of a single data center. This is due to the limited power su…
HybHuff: Lossless Compression for Hypergraphs via Entropy-Guided Huffman-Bitwise Coordination
Tianyu Zhao, Dongfang Zhao, Luanzheng Guo +1
Hypergraphs provide a natural representation for many-to-many relationships in data-intensive applications, yet their scalability is often hindered by high memory consumption. Whil…
SuperSAM: Crafting a SAM Supernetwork via Structured Pruning and Unstructured Parameter Prioritization
Waqwoya Abebe, Sadegh Jafari, Sixing Yu +6
Neural Architecture Search (NAS) is a powerful approach of automating the design of efficient neural architectures. In contrast to traditional NAS methods, recently proposed one-sh…
MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs
Aishwarya Sarkar, Sayan Ghosh, Nathan R. Tallent +1
Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significa…
Final Report for CHESS: Cloud, High-Performance Computing, and Edge for Science and Security
Nathan Tallent, Jan Strube, Luanzheng Guo +9
Automating the theory-experiment cycle requires effective distributed workflows that utilize a computing continuum spanning lab instruments, edge sensors, computing resources at mu…
Workflows Community Summit 2024: Future Trends and Challenges in Scientific Workflows
Rafael Ferreira da Silva, Deborah Bard, Kyle Chard +108
The Workflows Community Summit gathered 111 participants from 18 countries to discuss emerging trends and challenges in scientific workflows, focusing on six key areas: time-sensit…
Overcoming Memory Constraints in Quantum Circuit Simulation with a High-Fidelity Compression Framework
Boyuan Zhang, Bo Fang, Fanjiang Ye +4
Full-state quantum circuit simulation requires exponentially increased memory size to store the state vector as the number of qubits scales, presenting significant limitations in c…
Order-Preserving Dimension Reduction for Multimodal Semantic Embedding
Chengyu Gong, Gefei Shen, Luanzheng Guo +2
Searching for the -nearest neighbors (KNN) in multimodal data retrieval is computationally expensive, particularly due to the inherent difficulty in comparing similarity measure…
SAM-I-Am: Semantic Boosting for Zero-shot Atomic-Scale Electron Micrograph Segmentation
Waqwoya Abebe, Jan Strube, Luanzheng Guo +5
Image segmentation is a critical enabler for tasks ranging from medical diagnostics to autonomous driving. However, the correct segmentation semantics - where are boundaries locate…
Bit-GraphBLAS: Bit-Level Optimizations of Matrix-Centric Graph Processing on GPU
Jou-An Chen, Hsin-Hsuan Sung, Xipeng Shen +3
In a general graph data structure like an adjacency matrix, when edges are homogeneous, the connectivity of two nodes can be sufficiently represented using a single bit. This insig…
Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
Ang Li, Shuaiwen Leon Song, Jieyang Chen +4
High performance multi-GPU computing becomes an inevitable trend due to the ever-increasing demand on computation capability in emerging domains such as deep learning, big data and…