Topologically Densified Distributions
arXiv:2002.04805
Abstract
We study regularization in the context of small sample-size learning with over-parameterized neural networks. Specifically, we shift focus from architectural properties, such as norms on the network weights, to properties of the internal representations before a linear classifier. Specifically, we impose a topological constraint on samples drawn from the probability measure induced in that space. This provably leads to mass concentration effects around the representations of training instances, i.e., a property beneficial for generalization. By leveraging previous work to impose topological constraints in a neural network setting, we provide empirical evidence (across various vision benchmarks) to support our claim for better generalization.
References in corpus (14)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Opening the Black Box of Deep Neural Networks via Information
- Manifold Mixup: Better Representations by Interpolating Hidden States
- Robust Large Margin Deep Neural Networks
- L2 Regularization versus Batch and Weight Normalization
- Uniform convergence may be unable to explain generalization in deep learning
- A Kernel Theory of Modern Data Augmentation
- Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology
- On Characterizing the Capacity of Neural Networks using Algebraic Topology
- A Topological Regularizer for Classifiers via Persistent Homology
- Augment your batch: better training with larger batches
- Connectivity-Optimized Representation Learning via Persistent Homology
- Neural Networks Regularization Through Class-wise Invariant Representation Learning
- Regularizing activations in neural networks via distribution matching with the Wasserstein metric