Online Nonparametric Regression
arXiv:1402.2594
Abstract
We establish optimal rates for online regression for arbitrary classes of regression functions in terms of the sequential entropy introduced in (Rakhlin, Sridharan, Tewari, 2010). The optimal rates are shown to exhibit a phase transition analogous to the i.i.d./statistical learning case, studied in (Rakhlin, Sridharan, Tsybakov 2013). In the frequently encountered situation when sequential entropy and i.i.d. empirical entropy match, our results point to the interesting phenomenon that the rates for statistical learning with squared loss and online nonparametric regression are the same. In addition to a non-algorithmic study of minimax regret, we exhibit a generic forecaster that enjoys the established optimal rates. We also provide a recipe for designing online regression algorithms that can be computationally efficient. We illustrate the techniques by deriving existing and new forecasters for the case of finite experts and for online linear regression.
Cited by in corpus (11)
- Learning with Square Loss: Localization through Offset Rademacher Complexity
- Adaptive Online Learning
- Online Forecasting of Total-Variation-bounded Sequences
- Learning to Bid Optimally and Efficiently in Adversarial First-price Auctions
- Optimal Dynamic Regret in Exp-Concave Online Learning
- On Empirical Risk Minimization with Dependent and Heavy-Tailed Data
- Nonparametric Online Regression while Learning the Metric
- Towards a Dimension-Free Understanding of Adaptive Linear Control
- Adaptive Online Estimation of Piecewise Polynomial Trends
- Locally-Adaptive Nonparametric Online Learning
- Localization, Convexity, and Star Aggregation