DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

Abstract

We have witnessed rapid evolution of deep neural network architecture design in the past years. These latest progresses greatly facilitate the developments in various areas such as computer vision and natural language processing. However, along with the extraordinary performance, these state-of-the-art models also bring in expensive computational cost. Directly deploying these models into applications with real-time requirement is still infeasible. Recently, Hinton etal. have shown that the dark knowledge within a powerful teacher model can significantly help the training of a smaller and faster student network. These knowledge are vastly beneficial to improve the generalization ability of the student model. Inspired by their work, we introduce a new type of knowledge -- cross sample similarities for model compression and acceleration. This knowledge can be naturally derived from deep metric learning model. To transfer them, we bring the "learning to rank" technique into deep metric learning formulation. We test our proposed DarkRank method on various metric learning tasks including pedestrian re-identification, image retrieval and image clustering. The results are quite encouraging. Our method can improve over the baseline method by a large margin. Moreover, it is fully compatible with other existing methods. When combined, the performance can be further boosted.

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• In Defense of the Triplet Loss for Person Re-Identification
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• Binarized Neural Networks
• Large-Margin Softmax Loss for Convolutional Neural Networks
• Like What You Like: Knowledge Distill via Neuron Selectivity Transfer
• Unsupervised Learning of Visual Representations using Videos
• A Good Practice Towards Top Performance of Face Recognition: Transferred Deep Feature Fusion

Cited by in corpus (5)

• A Survey of Model Compression and Acceleration for Deep Neural Networks
• SphereReID: Deep Hypersphere Manifold Embedding for Person Re-Identification
• Dual Attention Matching Network for Context-Aware Feature Sequence based Person Re-Identification
• Knowledge Distillation with Adversarial Samples Supporting Decision Boundary
• Self-Regulation for Semantic Segmentation