Learning with Non-Convex Truncated Losses by SGD

Abstract

Learning with a {\it convex loss} function has been a dominating paradigm for many years. It remains an interesting question how non-convex loss functions help improve the generalization of learning with broad applicability. In this paper, we study a family of objective functions formed by truncating traditional loss functions, which is applicable to both shallow learning and deep learning. Truncating loss functions has potential to be less vulnerable and more robust to large noise in observations that could be adversarial. More importantly, it is a generic technique without assuming the knowledge of noise distribution. To justify non-convex learning with truncated losses, we establish excess risk bounds of empirical risk minimization based on truncated losses for heavy-tailed output, and statistical error of an approximate stationary point found by stochastic gradient descent (SGD) method. Our experiments for shallow and deep learning for regression with outliers, corrupted data and heavy-tailed noise further justify the proposed method.

References in corpus (6)

• High-dimensional regression with noisy and missing data: Provable guarantees with nonconvexity
• Loss minimization and parameter estimation with heavy tails
• A Hitting Time Analysis of Stochastic Gradient Langevin Dynamics
• Fast Rates for General Unbounded Loss Functions: from ERM to Generalized Bayes
• Learning with Square Loss: Localization through Offset Rademacher Complexity
• Fast learning rates with heavy-tailed losses

Cited by in corpus (10)

• Distributed Stochastic Consensus Optimization with Momentum for Nonconvex Nonsmooth Problems
• First-order Convergence Theory for Weakly-Convex-Weakly-Concave Min-max Problems
• Universal Stagewise Learning for Non-Convex Problems with Convergence on Averaged Solutions
• Stability and Generalization of Stochastic Gradient Methods for Minimax Problems
• On the Convergence of (Stochastic) Gradient Descent with Extrapolation for Non-Convex Optimization
• Stochastic Optimization for Non-convex Inf-Projection Problems
• Nearly Optimal Robust Method for Convex Compositional Problems with Heavy-Tailed Noise
• Stochastic Optimization for DC Functions and Non-smooth Non-convex Regularizers with Non-asymptotic Convergence
• Bandwidth-based Step-Sizes for Non-Convex Stochastic Optimization
• Faster Stochastic Quasi-Newton Methods