Learning Soft Labels via Meta Learning

Abstract

One-hot labels do not represent soft decision boundaries among concepts, and hence, models trained on them are prone to overfitting. Using soft labels as targets provide regularization, but different soft labels might be optimal at different stages of optimization. Also, training with fixed labels in the presence of noisy annotations leads to worse generalization. To address these limitations, we propose a framework, where we treat the labels as learnable parameters, and optimize them along with model parameters. The learned labels continuously adapt themselves to the model's state, thereby providing dynamic regularization. When applied to the task of supervised image-classification, our method leads to consistent gains across different datasets and architectures. For instance, dynamically learned labels improve ResNet18 by 2.1% on CIFAR100. When applied to dataset containing noisy labels, the learned labels correct the annotation mistakes, and improves over state-of-the-art by a significant margin. Finally, we show that learned labels capture semantic relationship between classes, and thereby improve teacher models for the downstream task of distillation.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Distilling the Knowledge in a Neural Network
• Focal Loss for Dense Object Detection
• When Does Label Smoothing Help?
• Learning to Reweight Examples for Robust Deep Learning
• Regularizing Neural Networks by Penalizing Confident Output Distributions
• Joint Optimization Framework for Learning with Noisy Labels
• Learning to Impute: A General Framework for Semi-supervised Learning

Cited by in corpus (4)

• Vicarious Offense and Noise Audit of Offensive Speech Classifiers: Unifying Human and Machine Disagreement on What is Offensive
• Faster Meta Update Strategy for Noise-Robust Deep Learning
• Generalized Data Weighting via Class-level Gradient Manipulation
• Exploiting Class Similarity for Machine Learning with Confidence Labels and Projective Loss Functions