Fast Rates by Transferring from Auxiliary Hypotheses

Abstract

In this work we consider the learning setting where, in addition to the training set, the learner receives a collection of auxiliary hypotheses originating from other tasks. We focus on a broad class of ERM-based linear algorithms that can be instantiated with any non-negative smooth loss function and any strongly convex regularizer. We establish generalization and excess risk bounds, showing that, if the algorithm is fed with a good combination of source hypotheses, generalization happens at the fast rate $\mathcal{O}(1/m)$ instead of the usual $\mathcal{O}(1/\sqrt{m})$. On the other hand, if the source hypotheses combination is a misfit for the target task, we recover the usual learning rate. As a byproduct of our study, we also prove a new bound on the Rademacher complexity of the smooth loss class under weaker assumptions compared to previous works.

References in corpus (3)

• Frustratingly Easy Domain Adaptation
• Theoretical Models of Learning to Learn
• One-Shot Adaptation of Supervised Deep Convolutional Models

Cited by in corpus (9)

• Information-Theoretic Generalization Bounds for Meta-Learning and Applications
• A survey on domain adaptation theory: learning bounds and theoretical guarantees
• A General Class of Transfer Learning Regression without Implementation Cost
• Hypothesis Disparity Regularized Mutual Information Maximization
• Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithm
• Transfer Learning via ${\ell }_1$ Regularization
• Resource and data efficient self supervised learning
• A Distribution-Dependent Analysis of Meta-Learning
• Adaptive Learning to Speed-Up Control of Prosthetic Hands: a Few Things Everybody Should Know