A Generalization Theory based on Independent and Task-Identically Distributed Assumption

Abstract

Existing generalization theories analyze the generalization performance mainly based on the model complexity and training process. The ignorance of the task properties, which results from the widely used IID assumption, makes these theories fail to interpret many generalization phenomena or guide practical learning tasks. In this paper, we propose a new Independent and Task-Identically Distributed (ITID) assumption, to consider the task properties into the data generating process. The derived generalization bound based on the ITID assumption identifies the significance of hypothesis invariance in guaranteeing generalization performance. Based on the new bound, we introduce a practical invariance enhancement algorithm from the perspective of modifying data distributions. Finally, we verify the algorithm and theorems in the context of image classification task on both toy and real-world datasets. The experimental results demonstrate the reasonableness of the ITID assumption and the effectiveness of new generalization theory in improving practical generalization performance.

14 pages, 8 figures

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• mixup: Beyond Empirical Risk Minimization
• Focal Loss for Dense Object Detection
• Understanding deep learning requires rethinking generalization
• Opening the Black Box of Deep Neural Networks via Information
• Random Erasing Data Augmentation
• AutoAugment: Learning Augmentation Policies from Data
• Visualizing and Understanding Convolutional Networks
• Train faster, generalize better: Stability of stochastic gradient descent
• Causally Regularized Learning with Agnostic Data Selection Bias
• Chromatic PAC-Bayes Bounds for Non-IID Data: Applications to Ranking and Stationary $\beta$-Mixing Processes
• A New Distance Measure for Non-Identical Data with Application to Image Classification