Derivative Manipulation for General Example Weighting
arXiv:1905.11233
Abstract
Real-world large-scale datasets usually contain noisy labels and are imbalanced. Therefore, we propose derivative manipulation (DM), a novel and general example weighting approach for training robust deep models under these adverse conditions. DM has two main merits. First, loss function and example weighting are common techniques in the literature. DM reveals their connection (a loss function does example weighting) and is a replacement of both. Second, despite that a loss defines an example weighting scheme by its derivative, in the loss design, we need to consider whether it is differentiable. Instead, DM is more flexible by directly modifying the derivative so that a loss can be a non-elementary format too. Technically, DM defines an emphasis density function by a derivative magnitude function. DM is generic in that diverse weighting schemes can be derived. Extensive experiments on both vision and language tasks prove DM's effectiveness.
References in corpus (25)
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- Distilling the Knowledge in a Neural Network
- Distributed Representations of Sentences and Documents
- Learning to Reweight Examples for Robust Deep Learning
- Training Deep Neural Networks on Noisy Labels with Bootstrapping
- Training Convolutional Networks with Noisy Labels
- Deep Learning is Robust to Massive Label Noise
- A Closer Look at Memorization in Deep Networks
- Dimensionality-Driven Learning with Noisy Labels
- Unsupervised Label Noise Modeling and Loss Correction
- Masking: A New Perspective of Noisy Supervision
- Online Batch Selection for Faster Training of Neural Networks
- Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels
- Joint Optimization Framework for Learning with Noisy Labels
- Variance Reduction in SGD by Distributed Importance Sampling
- What Objective Does Self-paced Learning Indeed Optimize?
- Quality Aware Network for Set to Set Recognition
- Robust Inference via Generative Classifiers for Handling Noisy Labels
- SIGUA: Forgetting May Make Learning with Noisy Labels More Robust
- On Symmetric Losses for Learning from Corrupted Labels
- Learning From Noisy Large-Scale Datasets With Minimal Supervision
- Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification
- IMAE for Noise-Robust Learning: Mean Absolute Error Does Not Treat Examples Equally and Gradient Magnitude's Variance Matters
- Heated-Up Softmax Embedding
- ProSelfLC: Progressive Self Label Correction for Training Robust Deep Neural Networks