Transfer Learning for High-dimensional Linear Regression: Prediction, Estimation, and Minimax Optimality

Abstract

This paper considers the estimation and prediction of a high-dimensional linear regression in the setting of transfer learning, using samples from the target model as well as auxiliary samples from different but possibly related regression models. When the set of "informative" auxiliary samples is known, an estimator and a predictor are proposed and their optimality is established. The optimal rates of convergence for prediction and estimation are faster than the corresponding rates without using the auxiliary samples. This implies that knowledge from the informative auxiliary samples can be transferred to improve the learning performance of the target problem. In the case that the set of informative auxiliary samples is unknown, we propose a data-driven procedure for transfer learning, called Trans-Lasso, and reveal its robustness to non-informative auxiliary samples and its efficiency in knowledge transfer. The proposed procedures are demonstrated in numerical studies and are applied to a dataset concerning the associations among gene expressions. It is shown that Trans-Lasso leads to improved performance in gene expression prediction in a target tissue by incorporating the data from multiple different tissues as auxiliary samples.

References in corpus (9)

• Nearly unbiased variable selection under minimax concave penalty
• Frustratingly Easy Domain Adaptation
• Taking Advantage of Sparsity in Multi-Task Learning
• Exponential Screening and optimal rates of sparse estimation
• Restricted Eigenvalue Conditions on Subgaussian Random Matrices
• High-dimensional inference in misspecified linear models
• Transfer Learning for Nonparametric Classification: Minimax Rate and Adaptive Classifier
• ExSIS: Extended Sure Independence Screening for Ultrahigh-dimensional Linear Models
• Adaptive Sparse Estimation with Side Information

Cited by in corpus (4)

• Minimax Estimation for Personalized Federated Learning: An Alternative between FedAvg and Local Training?
• Improving Adversarial Robustness via Unlabeled Out-of-Domain Data
• Optimal oracle inequalities for solving projected fixed-point equations
• Learning Optimal Distributionally Robust Individualized Treatment Rules