Efficient learning with robust gradient descent
arXiv:1706.00182
Abstract
Minimizing the empirical risk is a popular training strategy, but for learning tasks where the data may be noisy or heavy-tailed, one may require many observations in order to generalize well. To achieve better performance under less stringent requirements, we introduce a procedure which constructs a robust approximation of the risk gradient for use in an iterative learning routine. Using high-probability bounds on the excess risk of this algorithm, we show that our update does not deviate far from the ideal gradient-based update. Empirical tests using both controlled simulations and real-world benchmark data show that in diverse settings, the proposed procedure can learn more efficiently, using less resources (iterations and observations) while generalizing better.
References in corpus (7)
- Geometric median and robust estimation in Banach spaces
- Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent
- Loss minimization and parameter estimation with heavy tails
- Robust empirical mean Estimators
- Optimal Learners for Multiclass Problems
- Learning from MOM's principles: Le Cam's approach
- High confidence estimates of the mean of heavy-tailed real random variables