SVMax: A Feature Embedding Regularizer
arXiv:2103.02770
Abstract
A neural network regularizer (e.g., weight decay) boosts performance by explicitly penalizing the complexity of a network. In this paper, we penalize inferior network activations -- feature embeddings -- which in turn regularize the network's weights implicitly. We propose singular value maximization (SVMax) to learn a more uniform feature embedding. The SVMax regularizer supports both supervised and unsupervised learning. Our formulation mitigates model collapse and enables larger learning rates. We evaluate the SVMax regularizer using both retrieval and generative adversarial networks. We leverage a synthetic mixture of Gaussians dataset to evaluate SVMax in an unsupervised setting. For retrieval networks, SVMax achieves significant improvement margins across various ranking losses. Code available at https://bit.ly/3jNkgDt
References in corpus (20)
- Distilling the Knowledge in a Neural Network
- Bootstrap your own latent: A new approach to self-supervised Learning
- In Defense of the Triplet Loss for Person Re-Identification
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
- Momentum Contrast for Unsupervised Visual Representation Learning
- Adversarial Feature Learning
- Pruning Filters for Efficient ConvNets
- Contrastive Multiview Coding
- Exploring Simple Siamese Representation Learning
- Unrolled Generative Adversarial Networks
- Metric Learning with Adaptive Density Discrimination
- Training Deep Networks with Structured Layers by Matrix Backpropagation
- Robust Learning with Jacobian Regularization
- Triplet-Center Loss for Multi-View 3D Object Retrieval
- Mode Seeking Generative Adversarial Networks for Diverse Image Synthesis
- MGGAN: Solving Mode Collapse using Manifold Guided Training
- Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterization
- Trace norm regularization and faster inference for embedded speech recognition RNNs
- On regularization for a convolutional kernel in neural networks