Webly Supervised Image Classification with Metadata: Automatic Noisy Label Correction via Visual-Semantic Graph

Abstract

Webly supervised learning becomes attractive recently for its efficiency in data expansion without expensive human labeling. However, adopting search queries or hashtags as web labels of images for training brings massive noise that degrades the performance of DNNs. Especially, due to the semantic confusion of query words, the images retrieved by one query may contain tremendous images belonging to other concepts. For example, searching `tiger cat' on Flickr will return a dominating number of tiger images rather than the cat images. These realistic noisy samples usually have clear visual semantic clusters in the visual space that mislead DNNs from learning accurate semantic labels. To correct real-world noisy labels, expensive human annotations seem indispensable. Fortunately, we find that metadata can provide extra knowledge to discover clean web labels in a labor-free fashion, making it feasible to automatically provide correct semantic guidance among the massive label-noisy web data. In this paper, we propose an automatic label corrector VSGraph-LC based on the visual-semantic graph. VSGraph-LC starts from anchor selection referring to the semantic similarity between metadata and correct label concepts, and then propagates correct labels from anchors on a visual graph using graph neural network (GNN). Experiments on realistic webly supervised learning datasets Webvision-1000 and NUS-81-Web show the effectiveness and robustness of VSGraph-LC. Moreover, VSGraph-LC reveals its advantage on the open-set validation set.

Accepted to ACM Multimedia 2020 (Oral)

References in corpus (15)

• Semi-Supervised Classification with Graph Convolutional Networks
• mixup: Beyond Empirical Risk Minimization
• Graph Neural Networks: A Review of Methods and Applications
• Simplifying Graph Convolutional Networks
• The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale
• Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey
• Billion-scale semi-supervised learning for image classification
• WebVision Database: Visual Learning and Understanding from Web Data
• Noise Tolerance under Risk Minimization
• CNN-RNN: A Unified Framework for Multi-label Image Classification
• Robust Loss Functions under Label Noise for Deep Neural Networks
• Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning
• OCGAN: One-class Novelty Detection Using GANs with Constrained Latent Representations
• CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images
• Deep Self-Learning From Noisy Labels