Fine-grained Categorization and Dataset Bootstrapping using Deep Metric Learning with Humans in the Loop

Abstract

Existing fine-grained visual categorization methods often suffer from three challenges: lack of training data, large number of fine-grained categories, and high intraclass vs. low inter-class variance. In this work we propose a generic iterative framework for fine-grained categorization and dataset bootstrapping that handles these three challenges. Using deep metric learning with humans in the loop, we learn a low dimensional feature embedding with anchor points on manifolds for each category. These anchor points capture intra-class variances and remain discriminative between classes. In each round, images with high confidence scores from our model are sent to humans for labeling. By comparing with exemplar images, labelers mark each candidate image as either a "true positive" or a "false positive". True positives are added into our current dataset and false positives are regarded as "hard negatives" for our metric learning model. Then the model is retrained with an expanded dataset and hard negatives for the next round. To demonstrate the effectiveness of the proposed framework, we bootstrap a fine-grained flower dataset with 620 categories from Instagram images. The proposed deep metric learning scheme is evaluated on both our dataset and the CUB-200-2001 Birds dataset. Experimental evaluations show significant performance gain using dataset bootstrapping and demonstrate state-of-the-art results achieved by the proposed deep metric learning methods.

10 pages, 9 figures, CVPR 2016

References in corpus (9)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Caffe: Convolutional Architecture for Fast Feature Embedding
• Going Deeper with Convolutions
• LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
• Simultaneous Feature Learning and Hash Coding with Deep Neural Networks
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets
• Learning Fine-grained Image Similarity with Deep Ranking
• Bilinear CNNs for Fine-grained Visual Recognition
• Learning Visual Clothing Style with Heterogeneous Dyadic Co-occurrences

Cited by in corpus (7)

• Object-Part Attention Model for Fine-grained Image Classification
• Knowledge Concentration: Learning 100K Object Classifiers in a Single CNN
• Class Rectification Hard Mining for Imbalanced Deep Learning
• Hard-Aware Deeply Cascaded Embedding
• Learning a Discriminative Filter Bank within a CNN for Fine-grained Recognition
• Incorporating Intra-Class Variance to Fine-Grained Visual Recognition
• Low-rank Bilinear Pooling for Fine-Grained Classification