Heated-Up Softmax Embedding

Abstract

Metric learning aims at learning a distance which is consistent with the semantic meaning of the samples. The problem is generally solved by learning an embedding for each sample such that the embeddings of samples of the same category are compact while the embeddings of samples of different categories are spread-out in the feature space. We study the features extracted from the second last layer of a deep neural network based classifier trained with the cross entropy loss on top of the softmax layer. We show that training classifiers with different temperature values of softmax function leads to features with different levels of compactness. Leveraging these insights, we propose a "heating-up" strategy to train a classifier with increasing temperatures, leading the corresponding embeddings to achieve state-of-the-art performance on a variety of metric learning benchmarks.

arXiv comment11 pages, 4 figures

References in corpus (5)

• Distilling the Knowledge in a Neural Network
• CNN Features off-the-shelf: an Astounding Baseline for Recognition
• Learning Deep Embeddings with Histogram Loss
• Smart Mining for Deep Metric Learning
• Learning Spread-out Local Feature Descriptors

Cited by in corpus (18)

• Circle Loss: A Unified Perspective of Pair Similarity Optimization
• Understanding and Improving Knowledge Distillation
• Combination of Multiple Global Descriptors for Image Retrieval
• DER: Dynamically Expandable Representation for Class Incremental Learning
• Dynamic Dual-Attentive Aggregation Learning for Visible-Infrared Person Re-Identification
• Deep Metric Learning with Spherical Embedding
• Temperature as Uncertainty in Contrastive Learning
• Derivative Manipulation for General Example Weighting
• Improved Embeddings with Easy Positive Triplet Mining
• Instance Cross Entropy for Deep Metric Learning
• Transformer-based Language Model Fine-tuning Methods for COVID-19 Fake News Detection
• Contextual Temperature for Language Modeling
• Variational Metric Scaling for Metric-Based Meta-Learning
• Dynamic Spatial Verification for Large-Scale Object-Level Image Retrieval
• Text analysis and deep learning: A network approach
• Improving Calibration in Deep Metric Learning With Cross-Example Softmax
• Learning Natural Language Generation from Scratch
• QATM: Quality-Aware Template Matching For Deep Learning