Distributed Robust Learning

Abstract

We propose a framework for distributed robust statistical learning on {\em big contaminated data}. The Distributed Robust Learning (DRL) framework can reduce the computational time of traditional robust learning methods by several orders of magnitude. We analyze the robustness property of DRL, showing that DRL not only preserves the robustness of the base robust learning method, but also tolerates contaminations on a constant fraction of results from computing nodes (node failures). More precisely, even in presence of the most adversarial outlier distribution over computing nodes, DRL still achieves a breakdown point of at least ${\lambda }^{\ast }/2$, where ${\lambda }^{\ast }$ is the break down point of corresponding centralized algorithm. This is in stark contrast with naive division-and-averaging implementation, which may reduce the breakdown point by a factor of $k$ when $k$ computing nodes are used. We then specialize the DRL framework for two concrete cases: distributed robust principal component analysis and distributed robust regression. We demonstrate the efficiency and the robustness advantages of DRL through comprehensive simulations and predicting image tags on a large-scale image set.

18 pages, 2 figures

References in corpus (5)

• Caffe: Convolutional Architecture for Fast Feature Embedding
• High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity
• Geometric median and robust estimation in Banach spaces
• Robust PCA in High-dimension: A Deterministic Approach
• Comunication-Efficient Algorithms for Statistical Optimization

Cited by in corpus (19)

• Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
• Federated Variance-Reduced Stochastic Gradient Descent with Robustness to Byzantine Attacks
• Byzantine Stochastic Gradient Descent
• Robust Federated Learning in a Heterogeneous Environment
• Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
• Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation
• Attack-Resistant Federated Learning with Residual-based Reweighting
• Learning from History for Byzantine Robust Optimization
• Byzantine-Resilient Non-Convex Stochastic Gradient Descent
• Variance Reduced Median-of-Means Estimator for Byzantine-Robust Distributed Inference
• Defending Against Saddle Point Attack in Byzantine-Robust Distributed Learning
• Robust Training in High Dimensions via Block Coordinate Geometric Median Descent
• On the Sample Complexity of Adversarial Multi-Source PAC Learning
• Distributed Newton Can Communicate Less and Resist Byzantine Workers
• Holdout SGD: Byzantine Tolerant Federated Learning
• Zeno++: Robust Fully Asynchronous SGD
• Efficient Byzantine-Resilient Stochastic Gradient Desce
• Escaping Saddle Points in Distributed Newton's Method with Communication Efficiency and Byzantine Resilience
• Accelerated Stochastic ExtraGradient: Mixing Hessian and Gradient Similarity to Reduce Communication in Distributed and Federated Learning