354 citations · 1.1k across the 23 of their papers we have counts for
17 papers · 1 filter
Maestro: Uncovering Low-Rank Structures via Trainable Decomposition
Samuel Horvath, Stefanos Laskaridis, Shashank Rajput +1
Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. However, these models have been getting increasingly large as they become more accurate a…
Cuttlefish: Low-Rank Model Training without All the Tuning
Hongyi Wang, Saurabh Agarwal, Pongsakorn U-chupala +3
Recent research has shown that training low-rank neural networks can effectively reduce the total number of trainable parameters without sacrificing predictive accuracy, resulting…
Memory-adaptive Depth-wise Heterogeneous Federated Learning
Kai Zhang, Yutong Dai, Hongyi Wang +3
Federated learning is a promising paradigm that allows multiple clients to collaboratively train a model without sharing the local data. However, the presence of heterogeneous devi…
Federated Learning as Variational Inference: A Scalable Expectation Propagation Approach
Han Guo, Philip Greengard, Hongyi Wang +3
The canonical formulation of federated learning treats it as a distributed optimization problem where the model parameters are optimized against a global loss function that decompo…
Does compressing activations help model parallel training?
Song Bian, Dacheng Li, Hongyi Wang +2
Large-scale Transformer models are known for their exceptional performance in a range of tasks, but training them can be difficult due to the requirement for communication-intensiv…
MPCFormer: fast, performant and private Transformer inference with MPC
Dacheng Li, Rulin Shao, Hongyi Wang +3
Enabling private inference is crucial for many cloud inference services that are based on Transformer models. However, existing private inference solutions can increase the inferen…
AMP: Automatically Finding Model Parallel Strategies with Heterogeneity Awareness
Dacheng Li, Hongyi Wang, Eric Xing +1
Scaling up model sizes can lead to fundamentally new capabilities in many machine learning (ML) tasks. However, training big models requires strong distributed system expertise to…
Rare Gems: Finding Lottery Tickets at Initialization
Kartik Sreenivasan, Jy-yong Sohn, Liu Yang +6
Large neural networks can be pruned to a small fraction of their original size, with little loss in accuracy, by following a time-consuming "train, prune, re-train" approach. Frank…
Solon: Communication-efficient Byzantine-resilient Distributed Training via Redundant Gradients
Lingjiao Chen, Leshang Chen, Hongyi Wang +2
There has been a growing need to provide Byzantine-resilience in distributed model training. Existing robust distributed learning algorithms focus on developing sophisticated robus…
A Field Guide to Federated Optimization
Jianyu Wang, Zachary Charles, Zheng Xu +50
Federated learning and analytics are a distributed approach for collaboratively learning models (or statistics) from decentralized data, motivated by and designed for privacy prote…
Pufferfish: Communication-efficient Models At No Extra Cost
Hongyi Wang, Saurabh Agarwal, Dimitris Papailiopoulos
To mitigate communication overheads in distributed model training, several studies propose the use of compressed stochastic gradients, usually achieved by sparsification or quantiz…
Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification
Saurabh Agarwal, Hongyi Wang, Kangwook Lee +2
Distributed model training suffers from communication bottlenecks due to frequent model updates transmitted across compute nodes. To alleviate these bottlenecks, practitioners use…
Attack of the Tails: Yes, You Really Can Backdoor Federated Learning
Hongyi Wang, Kartik Sreenivasan, Shashank Rajput +5
Due to its decentralized nature, Federated Learning (FL) lends itself to adversarial attacks in the form of backdoors during training. The goal of a backdoor is to corrupt the perf…
FedML: A Research Library and Benchmark for Federated Machine Learning
Chaoyang He, Songze Li, Jinhyun So +17
Federated learning (FL) is a rapidly growing research field in machine learning. However, existing FL libraries cannot adequately support diverse algorithmic development; inconsist…
Federated Learning with Matched Averaging
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun +2
Federated learning allows edge devices to collaboratively learn a shared model while keeping the training data on device, decoupling the ability to do model training from the need…
DETOX: A Redundancy-based Framework for Faster and More Robust Gradient Aggregation
Shashank Rajput, Hongyi Wang, Zachary Charles +1
To improve the resilience of distributed training to worst-case, or Byzantine node failures, several recent approaches have replaced gradient averaging with robust aggregation meth…
ErasureHead: Distributed Gradient Descent without Delays Using Approximate Gradient Coding
Hongyi Wang, Zachary Charles, Dimitris Papailiopoulos
We present ErasureHead, a new approach for distributed gradient descent (GD) that mitigates system delays by employing approximate gradient coding. Gradient coded distributed GD us…