67 citations · 128 across the 14 of their papers we have counts for
7 papers · 1 filter
Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Models
Zhiquan Lai, Shengwei Li, Xudong Tang +5
Foundation models are becoming the dominant deep learning technologies. Pretraining a foundation model is always time-consumed due to the large scale of both the model parameter an…
DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation
Yu Tang, Chenyu Wang, Yufan Zhang +5
The further development of deep neural networks is hampered by the limited GPU memory resource. Therefore, the optimization of GPU memory resources is highly demanded. Swapping and…
Inertial Proximal Deep Learning Alternating Minimization for Efficient Neutral Network Training
Linbo Qiao, Tao Sun, Hengyue Pan +1
In recent years, the Deep Learning Alternating Minimization (DLAM), which is actually the alternating minimization applied to the penalty form of the deep neutral networks training…
P-ADMMiRNN: Training RNN with Stable Convergence via An Efficient and Paralleled ADMM Approach
Yu Tang, Zhigang Kan, Dequan Sun +4
It is hard to train Recurrent Neural Network (RNN) with stable convergence and avoid gradient vanishing and exploding problems, as the weights in the recurrent unit are repeated fr…
On the Iteration Complexity Analysis of Stochastic Primal-Dual Hybrid Gradient Approach with High Probability
Linbo Qiao, Tianyi Lin, Qi Qin +1
In this paper, we propose a stochastic Primal-Dual Hybrid Gradient (PDHG) approach for solving a wide spectrum of regularized stochastic minimization problems, where the regulariza…
An Incremental Path-Following Splitting Method for Linearly Constrained Nonconvex Nonsmooth Programs
Linbo Qiao, Wei Liu, Steven Hoi
The stationary point of Problem 2 is NOT the stationary point of Problem 1. We are sorry and we are working on fixing this error.
Stochastic Primal-Dual Proximal ExtraGradient Descent for Compositely Regularized Optimization
Tianyi Lin, Linbo Qiao, Teng Zhang +2
We consider a wide range of regularized stochastic minimization problems with two regularization terms, one of which is composed with a linear function. This optimization model abs…