Learning Deep Neural Networks under Agnostic Corrupted Supervision

Abstract

Training deep neural models in the presence of corrupted supervision is challenging as the corrupted data points may significantly impact the generalization performance. To alleviate this problem, we present an efficient robust algorithm that achieves strong guarantees without any assumption on the type of corruption and provides a unified framework for both classification and regression problems. Unlike many existing approaches that quantify the quality of the data points (e.g., based on their individual loss values), and filter them accordingly, the proposed algorithm focuses on controlling the collective impact of data points on the average gradient. Even when a corrupted data point failed to be excluded by our algorithm, the data point will have a very limited impact on the overall loss, as compared with state-of-the-art filtering methods based on loss values. Extensive experiments on multiple benchmark datasets have demonstrated the robustness of our algorithm under different types of corruption.

References in corpus (11)

• Training Deep Neural Networks on Noisy Labels with Bootstrapping
• Convergence Rates of Inexact Proximal-Gradient Methods for Convex Optimization
• Sever: A Robust Meta-Algorithm for Stochastic Optimization
• Robust Regression via Hard Thresholding
• Error-Bounded Correction of Noisy Labels
• Efficient Per-Example Gradient Computations
• On the Convergence of SGD with Biased Gradients
• Biased Stochastic First-Order Methods for Conditional Stochastic Optimization and Applications in Meta Learning
• Choosing the Sample with Lowest Loss makes SGD Robust
• High-Dimensional Robust Mean Estimation via Gradient Descent
• Robust Collaborative Learning with Noisy Labels