Beyond Least-Squares: Fast Rates for Regularized Empirical Risk Minimization through Self-Concordance

Abstract

We consider learning methods based on the regularization of a convex empirical risk by a squared Hilbertian norm, a setting that includes linear predictors and non-linear predictors through positive-definite kernels. In order to go beyond the generic analysis leading to convergence rates of the excess risk as $O(1/\sqrt{n})$ from $n$ observations, we assume that the individual losses are self-concordant, that is, their third-order derivatives are bounded by their second-order derivatives. This setting includes least-squares, as well as all generalized linear models such as logistic and softmax regression. For this class of losses, we provide a bias-variance decomposition and show that the assumptions commonly made in least-squares regression, such as the source and capacity conditions, can be adapted to obtain fast non-asymptotic rates of convergence by improving the bias terms, the variance terms or both.

Cited by in corpus (10)

• Kernel methods through the roof: handling billions of points efficiently
• Implicit Regularization of Random Feature Models
• Efficient improper learning for online logistic regression
• Fast and Furious Convergence: Stochastic Second Order Methods under Interpolation
• Globally Convergent Newton Methods for Ill-conditioned Generalized Self-concordant Losses
• A Spectral Analysis of Dot-product Kernels
• Self-Concordant Analysis of Frank-Wolfe Algorithms
• Fast rates in structured prediction
• Localization, Convexity, and Star Aggregation
• Near-Optimal Procedures for Model Discrimination with Non-Disclosure Properties