Self-Supervised Video Hashing with Hierarchical Binary Auto-encoder

Abstract

Existing video hash functions are built on three isolated stages: frame pooling, relaxed learning, and binarization, which have not adequately explored the temporal order of video frames in a joint binary optimization model, resulting in severe information loss. In this paper, we propose a novel unsupervised video hashing framework dubbed Self-Supervised Video Hashing (SSVH), that is able to capture the temporal nature of videos in an end-to-end learning-to-hash fashion. We specifically address two central problems: 1) how to design an encoder-decoder architecture to generate binary codes for videos; and 2) how to equip the binary codes with the ability of accurate video retrieval. We design a hierarchical binary autoencoder to model the temporal dependencies in videos with multiple granularities, and embed the videos into binary codes with less computations than the stacked architecture. Then, we encourage the binary codes to simultaneously reconstruct the visual content and neighborhood structure of the videos. Experiments on two real-world datasets (FCVID and YFCC) show that our SSVH method can significantly outperform the state-of-the-art methods and achieve the currently best performance on the task of unsupervised video retrieval.

References in corpus (12)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• YFCC100M: The New Data in Multimedia Research
• Exploiting Feature and Class Relationships in Video Categorization with Regularized Deep Neural Networks
• Image Captioning with Semantic Attention
• Adversarial Learning for Neural Dialogue Generation
• Sequence to Sequence -- Video to Text
• Feature Learning based Deep Supervised Hashing with Pairwise Labels
• HashNet: Deep Learning to Hash by Continuation
• Massive Exploration of Neural Machine Translation Architectures
• Hierarchical LSTM with Adjusted Temporal Attention for Video Captioning
• Deep Region Hashing for Efficient Large-scale Instance Search from Images

Cited by in corpus (17)

• Deep Recurrent Quantization for Generating Sequential Binary Codes
• Beyond Product Quantization: Deep Progressive Quantization for Image Retrieval
• Adaptive Exploration for Unsupervised Person Re-Identification
• Semi-Heterogeneous Three-Way Joint Embedding Network for Sketch-Based Image Retrieval
• DnS: Distill-and-Select for Efficient and Accurate Video Indexing and Retrieval
• CHAIN: Exploring Global-Local Spatio-Temporal Information for Improved Self-Supervised Video Hashing
• COCO-CN for Cross-Lingual Image Tagging, Captioning and Retrieval
• Live Laparoscopic Video Retrieval with Compressed Uncertainty
• From Selective Deep Convolutional Features to Compact Binary Representations for Image Retrieval
• One-shot Scene Graph Generation
• Self-supervised Video Retrieval Transformer Network
• Targeted Attack for Deep Hashing based Retrieval
• Decomposing Normal and Abnormal Features of Medical Images into Discrete Latent Codes for Content-Based Image Retrieval
• Long-Short Temporal Contrastive Learning of Video Transformers
• Encode the Unseen: Predictive Video Hashing for Scalable Mid-Stream Retrieval
• Video Segment Copy Detection Using Memory Constrained Hierarchical Batch-Normalized LSTM Autoencoder
• Fine-Grained Instance-Level Sketch-Based Video Retrieval