6 papers
MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations
Jiaxiang Zou, Yonghao Chen, Ruilong Wu +1
As large language models continue to scale, fine-grained block-scaled low-precision formats such as NVFP4 are increasingly adopted for their substantial throughput and memory benef…
ViFusion: In-Network Tensor Fusion for Scalable Video Feature Indexing
Yisu Wang, Yixiang Zhu, Xinjiao Li +3
Large-scale video feature indexing in datacenters is critically dependent on efficient data transfer. Although in-network computation has emerged as a compelling strategy for accel…
NetSenseML: Network-Adaptive Compression for Efficient Distributed Machine Learning
Yisu Wang, Xinjiao Li, Ruilong Wu +2
Training large-scale distributed machine learning models imposes considerable demands on network infrastructure, often resulting in sudden traffic spikes that lead to congestion, i…
Rethinking Dynamic Networks and Heterogeneous Computing with Automatic Parallelization
Ruilong Wu, Xinjiao Li, Yisu Wang +2
Hybrid parallelism techniques are essential for efficiently training large language models (LLMs). Nevertheless, current automatic parallel planning frameworks often overlook the s…
PacTrain: Pruning and Adaptive Sparse Gradient Compression for Efficient Collective Communication in Distributed Deep Learning
Yisu Wang, Ruilong Wu, Xinjiao Li +1
Large-scale deep neural networks (DNN) exhibit excellent performance for various tasks. As DNNs and datasets grow, distributed training becomes extremely time-consuming and demands…
Affordable HPC: Leveraging Small Clusters for Big Data and Graph Computing
Ruilong Wu, Yisu Wang, Dirk Kutscher
This study explores strategies for academic researchers to optimize computational resources within limited budgets, focusing on building small, efficient computing clusters. It del…