12 citations · 21 across the 10 of their papers we have counts for
10 papers
DreamDDP: Accelerating Data Parallel Distributed LLM Training with Layer-wise Scheduled Partial Synchronization
Zhenheng Tang, Zichen Tang, Junlin Huang +7
The growth of large language models (LLMs) increases challenges of accelerating distributed training across multiple GPUs in different data centers. Moreover, concerns about data p…
SphereFusion: Efficient Panorama Depth Estimation via Gated Fusion
Qingsong Yan, Qiang Wang, Kaiyong Zhao +4
Due to the rapid development of panorama cameras, the task of estimating panorama depth has attracted significant attention from the computer vision community, especially in applic…
FSMoE: A Flexible and Scalable Training System for Sparse Mixture-of-Experts Models
Xinglin Pan, Wenxiang Lin, Lin Zhang +5
Recent large language models (LLMs) have tended to leverage sparsity to reduce computations, employing the sparsely activated mixture-of-experts (MoE) technique. MoE introduces fou…
FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression
Zhenheng Tang, Xueze Kang, Yiming Yin +11
To alleviate hardware scarcity in training large deep neural networks (DNNs), particularly large language models (LLMs), we present FusionLLM, a decentralized training system desig…
Parm: Efficient Training of Large Sparsely-Activated Models with Dedicated Schedules
Xinglin Pan, Wenxiang Lin, Shaohuai Shi +3
Sparsely-activated Mixture-of-Expert (MoE) layers have found practical applications in enlarging the model size of large-scale foundation models, with only a sub-linear increase in…
CF-NeRF: Camera Parameter Free Neural Radiance Fields with Incremental Learning
Qingsong Yan, Qiang Wang, Kaiyong Zhao +4
Neural Radiance Fields (NeRF) have demonstrated impressive performance in novel view synthesis. However, NeRF and most of its variants still rely on traditional complex pipelines t…
Rethinking Disparity: A Depth Range Free Multi-View Stereo Based on Disparity
Qingsong Yan, Qiang Wang, Kaiyong Zhao +3
Existing learning-based multi-view stereo (MVS) methods rely on the depth range to build the 3D cost volume and may fail when the range is too large or unreliable. To address this…
SphereDepth: Panorama Depth Estimation from Spherical Domain
Qingsong Yan, Qiang Wang, Kaiyong Zhao +3
The panorama image can simultaneously demonstrate complete information of the surrounding environment and has many advantages in virtual tourism, games, robotics, etc. However, the…
A Quantitative Survey of Communication Optimizations in Distributed Deep Learning
Shaohuai Shi, Zhenheng Tang, Xiaowen Chu +3
Nowadays, large and complex deep learning (DL) models are increasingly trained in a distributed manner across multiple worker machines, in which extensive communications between wo…
MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms
Shaohuai Shi, Xiaowen Chu, Bo Li
Distributed synchronous stochastic gradient descent has been widely used to train deep neural networks on computer clusters. With the increase of computational power, network commu…