Towards Optimal Problem Dependent Generalization Error Bounds in Statistical Learning Theory

Abstract

We study problem-dependent rates, i.e., generalization errors that scale near-optimally with the variance, the effective loss, or the gradient norms evaluated at the "best hypothesis." We introduce a principled framework dubbed "uniform localized convergence," and characterize sharp problem-dependent rates for central statistical learning problems. From a methodological viewpoint, our framework resolves several fundamental limitations of existing uniform convergence and localization analysis approaches. It also provides improvements and some level of unification in the study of localized complexities, one-sided uniform inequalities, and sample-based iterative algorithms. In the so-called "slow rate" regime, we provides the first (moment-penalized) estimator that achieves the optimal variance-dependent rate for general "rich" classes; we also establish improved loss-dependent rate for standard empirical risk minimization. In the "fast rate" regime, we establish finite-sample problem-dependent bounds that are comparable to precise asymptotics. In addition, we show that iterative algorithms like gradient descent and first-order Expectation-Maximization can achieve optimal generalization error in several representative problems across the areas of non-convex learning, stochastic optimization, and learning with missing data.

References in corpus (19)

• SLOPE - Adaptive variable selection via convex optimization
• Robust Wasserstein Profile Inference and Applications to Machine Learning
• Risk bounds for statistical learning
• Empirical Bernstein Bounds and Sample Variance Penalization
• Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optima
• Regime Change: Bit-Depth versus Measurement-Rate in Compressive Sensing
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• Concentration inequalities and asymptotic results for ratio type empirical processes
• Uniform convergence may be unable to explain generalization in deep learning
• An Alternative View: When Does SGD Escape Local Minima?
• Logistic Regression: Tight Bounds for Stochastic and Online Optimization
• High Dimensional Expectation-Maximization Algorithm: Statistical Optimization and Asymptotic Normality
• Learning with Square Loss: Localization through Offset Rademacher Complexity
• SGD Converges to Global Minimum in Deep Learning via Star-convex Path
• Failures of model-dependent generalization bounds for least-norm interpolation
• Robust covariance estimation under $L_4-L_2$ norm equivalence
• On Uniform Convergence and Low-Norm Interpolation Learning
• Optimal learning via local entropies and sample compression
• An optimal unrestricted learning procedure

Cited by in corpus (1)

• Improved Learning Rates for Stochastic Optimization