Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy
arXiv:1911.09307
Abstract
Regularization plays a crucial role in machine learning models, especially for deep neural networks. The existing regularization techniques mainly rely on the i.i.d. assumption and only consider the knowledge from the current sample, without the leverage of the neighboring relationship between samples. In this work, we propose a general regularizer called \textbf{Patch-level Neighborhood Interpolation~(Pani)} that conducts a non-local representation in the computation of networks. Our proposal explicitly constructs patch-level graphs in different layers and then linearly interpolates neighborhood patch features, serving as a general and effective regularization strategy. Further, we customize our approach into two kinds of popular regularization methods, namely Virtual Adversarial Training (VAT) and MixUp as well as its variants. The first derived \textbf{Pani VAT} presents a novel way to construct non-local adversarial smoothness by employing patch-level interpolated perturbations. The second derived \textbf{Pani MixUp} method extends the MixUp, and achieves superiority over MixUp and competitive performance over state-of-the-art variants of MixUp method with a significant advantage in computational efficiency. Extensive experiments have verified the effectiveness of our Pani approach in both supervised and semi-supervised settings.
Accepted in ACML 2023 conference track
References in corpus (10)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Improving neural networks by preventing co-adaptation of feature detectors
- mixup: Beyond Empirical Risk Minimization
- Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
- Temporal Ensembling for Semi-Supervised Learning
- Theoretically Principled Trade-off between Robustness and Accuracy
- Triple Generative Adversarial Nets
- Semi-Supervised Learning via Compact Latent Space Clustering
- Smooth Neighbors on Teacher Graphs for Semi-supervised Learning
- Semi-Supervised Learning with GANs: Revisiting Manifold Regularization