A Kernel Theory of Modern Data Augmentation

Abstract

Data augmentation, a technique in which a training set is expanded with class-preserving transformations, is ubiquitous in modern machine learning pipelines. In this paper, we seek to establish a theoretical framework for understanding data augmentation. We approach this from two directions: First, we provide a general model of augmentation as a Markov process, and show that kernels appear naturally with respect to this model, even when we do not employ kernel classification. Next, we analyze more directly the effect of augmentation on kernel classifiers, showing that data augmentation can be approximated by first-order feature averaging and second-order variance regularization components. These frameworks both serve to illustrate the ways in which data augmentation affects the downstream learning model, and the resulting analyses provide novel connections between prior work in invariant kernels, tangent propagation, and robust optimization. Finally, we provide several proof-of-concept applications showing that our theory can be useful for accelerating machine learning workflows, such as reducing the amount of computation needed to train using augmented data, and predicting the utility of a transformation prior to training.

Cited by in corpus (28)

• On the Opportunities and Risks of Foundation Models
• Models Genesis
• Incorporating Symmetry into Deep Dynamics Models for Improved Generalization
• Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style
• Affinity and Diversity: Quantifying Mechanisms of Data Augmentation
• Model Patching: Closing the Subgroup Performance Gap with Data Augmentation
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data
• Neural Kernels Without Tangents
• Learning Invariances in Neural Networks
• Reducing the Model Variance of a Rectal Cancer Segmentation Network
• On the Benefits of Invariance in Neural Networks
• Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Network
• WeMix: How to Better Utilize Data Augmentation
• On the Generalization Effects of Linear Transformations in Data Augmentation
• A Fourier-based Framework for Domain Generalization
• On Interaction Between Augmentations and Corruptions in Natural Corruption Robustness
• Implicit Rugosity Regularization via Data Augmentation
• ATD: Augmenting CP Tensor Decomposition by Self Supervision
• Data augmentation in Bayesian neural networks and the cold posterior effect
• Restyling Data: Application to Unsupervised Domain Adaptation
• How Data Augmentation affects Optimization for Linear Regression
• Do CNNs Encode Data Augmentations?
• Building Legal Datasets
• Joint Text and Label Generation for Spoken Language Understanding
• Topologically Densified Distributions
• Noisy Feature Mixup
• Metadata Shaping: Natural Language Annotations for the Tail