208 citations · 326 across the 25 of their papers we have counts for
14 papers · 1 filter
Compute-Optimal Is Not Cluster-Optimal: Systems-Aware Scaling for Sparse Mixture-of-Experts
Soumajyoti Sarkar, Yuxin Tang, Sheng Zha
In large-scale pretraining, the algorithm, architecture, and systems decisions are conventionally made in disconnected stages. A scaling law stage selects an architecture and train…
Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning
Soumajyoti Sarkar, Leonard Lausen, Volkan Cevher +3
Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. These models use conditionally activated feedforward subnetworks…
Pre-training Differentially Private Models with Limited Public Data
Zhiqi Bu, Xinwei Zhang, Mingyi Hong +2
The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that…
Zero redundancy distributed learning with differential privacy
Zhiqi Bu, Justin Chiu, Ruixuan Liu +2
Deep learning using large models have achieved great success in a wide range of domains. However, training these models on billions of parameters is very challenging in terms of th…
On the accuracy and efficiency of group-wise clipping in differentially private optimization
Zhiqi Bu, Ruixuan Liu, Yu-Xiang Wang +2
Recent advances have substantially improved the accuracy, memory cost, and training speed of differentially private (DP) deep learning, especially on large vision and language mode…
Coupling public and private gradient provably helps optimization
Ruixuan Liu, Zhiqi Bu, Yu-xiang Wang +2
The success of large neural networks is crucially determined by the availability of data. It has been observed that training only on a small amount of public data, or privately on…
HYTREL: Hypergraph-enhanced Tabular Data Representation Learning
Pei Chen, Soumajyoti Sarkar, Leonard Lausen +4
Language models pretrained on large collections of tabular data have demonstrated their effectiveness in several downstream tasks. However, many of these models do not take into ac…
Large Language Models of Code Fail at Completing Code with Potential Bugs
Tuan Dinh, Jinman Zhao, Samson Tan +4
Large language models of code (Code-LLMs) have recently brought tremendous advances to code completion, a fundamental feature of programming assistance and code intelligence. Howev…
Differentially Private Optimization on Large Model at Small Cost
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha +1
Differentially private (DP) optimization is the standard paradigm to learn large neural networks that are accurate and privacy-preserving. The computational cost for DP deep learni…
Differentially Private Bias-Term Fine-tuning of Foundation Models
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha +1
We study the problem of differentially private (DP) fine-tuning of large pre-trained models -- a recent privacy-preserving approach suitable for solving downstream tasks with sensi…
Automatic Clipping: Differentially Private Deep Learning Made Easier and Stronger
Zhiqi Bu, Yu-Xiang Wang, Sheng Zha +1
Per-example gradient clipping is a key algorithmic step that enables practical differential private (DP) training for deep learning models. The choice of clipping threshold R, howe…
Accelerated Large Batch Optimization of BERT Pretraining in 54 minutes
Shuai Zheng, Haibin Lin, Sheng Zha +1
BERT has recently attracted a lot of attention in natural language understanding (NLU) and achieved state-of-the-art results in various NLU tasks. However, its success requires lar…
GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
Jian Guo, He He, Tong He +13
We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-…
Dynamic Mini-batch SGD for Elastic Distributed Training: Learning in the Limbo of Resources
Haibin Lin, Hang Zhang, Yifei Ma +4
With an increasing demand for training powers for deep learning algorithms and the rapid growth of computation resources in data centers, it is desirable to dynamically schedule di…