Understanding the role of importance weighting for deep learning

Abstract

The recent paper by Byrd & Lipton (2019), based on empirical observations, raises a major concern on the impact of importance weighting for the over-parameterized deep learning models. They observe that as long as the model can separate the training data, the impact of importance weighting diminishes as the training proceeds. Nevertheless, there lacks a rigorous characterization of this phenomenon. In this paper, we provide formal characterizations and theoretical justifications on the role of importance weighting with respect to the implicit bias of gradient descent and margin-based learning theory. We reveal both the optimization dynamics and generalization performance under deep learning models. Our work not only explains the various novel phenomenons observed for importance weighting in deep learning, but also extends to the studies where the weights are being optimized as part of the model, which applies to a number of topics under active research.

References in corpus (8)

• Distilling the Knowledge in a Neural Network
• Towards Understanding the Role of Over-Parametrization in Generalization of Neural Networks
• Counterfactual Risk Minimization: Learning from Logged Bandit Feedback
• Gradient descent aligns the layers of deep linear networks
• Gradient Descent Maximizes the Margin of Homogeneous Neural Networks
• Risk and parameter convergence of logistic regression
• Regularized Learning for Domain Adaptation under Label Shifts
• Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Models

Cited by in corpus (7)

• Self-supervised Learning is More Robust to Dataset Imbalance
• Enhanced Imitation Learning of Model Predictive Control through Importance Weighting
• SurvITE: Learning Heterogeneous Treatment Effects from Time-to-Event Data
• When is Importance Weighting Correction Needed for Covariate Shift Adaptation?
• Tutorial: Modern Theoretical Tools for Understanding and Designing Next-generation Information Retrieval System
• Leveraging Local Variation in Data: Sampling and Weighting Schemes for Supervised Deep Learning
• DIVA: Dataset Derivative of a Learning Task