399 citations · 1.5k across the 18 of their papers we have counts for
19 papers
Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k
Zangwei Zheng, Xiangyu Peng, Yuxuan Lou +30
Video generation models have achieved remarkable progress in the past year. The quality of AI video continues to improve, but at the cost of larger model size, increased data quant…
E^2VTS: Energy-Efficient Video Text Spotting from Unmanned Aerial Vehicles
Zhenyu Hu, Zhenyu Wu, Pengcheng Pi +6
Unmanned Aerial Vehicles (UAVs) based video text spotting has been extensively used in civil and military domains. UAV's limited battery capacity motivates us to develop an energy-…
Persia: An Open, Hybrid System Scaling Deep Learning-based Recommenders up to 100 Trillion Parameters
Xiangru Lian, Binhang Yuan, Xuefeng Zhu +24
Deep learning based models have dominated the current landscape of production recommender systems. Furthermore, recent years have witnessed an exponential growth of the model scale…
BAGUA: Scaling up Distributed Learning with System Relaxations
Shaoduo Gan, Xiangru Lian, Rui Wang +11
Recent years have witnessed a growing list of systems for distributed data-parallel training. Existing systems largely fit into two paradigms, i.e., parameter server and MPI-style…
DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning
Daochen Zha, Jingru Xie, Wenye Ma +4
Games are abstractions of the real world, where artificial agents learn to compete and cooperate with other agents. While significant achievements have been made in various perfect…
1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
Hanlin Tang, Shaoduo Gan, Ammar Ahmad Awan +6
Scalable training of large models (like BERT and GPT-3) requires careful optimization rooted in model design, architecture, and system capabilities. From a system standpoint, commu…
APMSqueeze: A Communication Efficient Adam-Preconditioned Momentum SGD Algorithm
Hanlin Tang, Shaoduo Gan, Samyam Rajbhandari +4
Adam is the important optimization algorithm to guarantee efficiency and accuracy for training many important tasks such as BERT and ImageNet. However, Adam is generally not compat…
Stochastic Recursive Momentum for Policy Gradient Methods
Huizhuo Yuan, Xiangru Lian, Ji Liu +1
In this paper, we propose a novel algorithm named STOchastic Recursive Momentum for Policy Gradient (STORM-PG), which operates a SARAH-type stochastic recursive variance-reduced po…
Stochastic Recursive Variance Reduction for Efficient Smooth Non-Convex Compositional Optimization
Huizhuo Yuan, Xiangru Lian, Ji Liu
Stochastic compositional optimization arises in many important machine learning tasks such as value function evaluation in reinforcement learning and portfolio management. The obje…
: Decentralization Meets Error-Compensated Compression
Hanlin Tang, Xiangru Lian, Shuang Qiu +4
Communication is a key bottleneck in distributed training. Recently, an \emph{error-compensated} compression technology was particularly designed for the \emph{centralized} learnin…
DoubleSqueeze: Parallel Stochastic Gradient Descent with Double-Pass Error-Compensated Compression
Hanlin Tang, Xiangru Lian, Chen Yu +2
A standard approach in large scale machine learning is distributed stochastic gradient training, which requires the computation of aggregated stochastic gradients over multiple nod…
Revisit Batch Normalization: New Understanding from an Optimization View and a Refinement via Composition Optimization
Xiangru Lian, Ji Liu
Batch Normalization (BN) has been used extensively in deep learning to achieve faster training process and better resulting models. However, whether BN works strongly depends on ho…
D: Decentralized Training over Decentralized Data
Hanlin Tang, Xiangru Lian, Ming Yan +2
While training a machine learning model using multiple workers, each of which collects data from their own data sources, it would be most useful when the data collected from differ…
Asynchronous Decentralized Parallel Stochastic Gradient Descent
Xiangru Lian, Wei Zhang, Ce Zhang +1
Most commonly used distributed machine learning systems are either synchronous or centralized asynchronous. Synchronous algorithms like AllReduce-SGD perform poorly in a heterogene…
Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
Xiangru Lian, Ce Zhang, Huan Zhang +3
Most distributed machine learning systems nowadays, including TensorFlow and CNTK, are built in a centralized fashion. One bottleneck of centralized algorithms lies on high communi…
Finite-sum Composition Optimization via Variance Reduced Gradient Descent
Xiangru Lian, Mengdi Wang, Ji Liu
The stochastic composition optimization proposed recently by Wang et al. [2014] minimizes the objective with the compositional expectation form: $\min_x~(\mathbb{E}_iF_i \circ \mat…
A Comprehensive Linear Speedup Analysis for Asynchronous Stochastic Parallel Optimization from Zeroth-Order to First-Order
Xiangru Lian, Huan Zhang, Cho-Jui Hsieh +2
Asynchronous parallel optimization received substantial successes and extensive attention recently. One of core theoretical questions is how much speedup (or benefit) the asynchron…
Staleness-aware Async-SGD for Distributed Deep Learning
Wei Zhang, Suyog Gupta, Xiangru Lian +1
Deep neural networks have been shown to achieve state-of-the-art performance in several machine learning tasks. Stochastic Gradient Descent (SGD) is the preferred optimization algo…
Asynchronous Parallel Stochastic Gradient for Nonconvex Optimization
Xiangru Lian, Yijun Huang, Yuncheng Li +1
Asynchronous parallel implementations of stochastic gradient (SG) have been broadly used in solving deep neural network and received many successes in practice recently. However, e…