Feature Re-Learning with Data Augmentation for Video Relevance Prediction

Abstract

Predicting the relevance between two given videos with respect to their visual content is a key component for content-based video recommendation and retrieval. Thanks to the increasing availability of pre-trained image and video convolutional neural network models, deep visual features are widely used for video content representation. However, as how two videos are relevant is task-dependent, such off-the-shelf features are not always optimal for all tasks. Moreover, due to varied concerns including copyright, privacy and security, one might have access to only pre-computed video features rather than original videos. We propose in this paper feature re-learning for improving video relevance prediction, with no need of revisiting the original video content. In particular, re-learning is realized by projecting a given deep feature into a new space by an affine transformation. We optimize the re-learning process by a novel negative-enhanced triplet ranking loss. In order to generate more training data, we propose a new data augmentation strategy which works directly on frame-level and video-level features. Extensive experiments in the context of the Hulu Content-based Video Relevance Prediction Challenge 2018 justify the effectiveness of the proposed method and its state-of-the-art performance for content-based video relevance prediction.

accepted by IEEE Transactions on Knowledge and Data Engineering (TKDE)

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• FaceNet: A Unified Embedding for Face Recognition and Clustering
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• End to End Learning for Self-Driving Cars
• The Effectiveness of Data Augmentation in Image Classification using Deep Learning
• YouTube-8M: A Large-Scale Video Classification Benchmark
• NAIS: Neural Attentive Item Similarity Model for Recommendation
• VSE++: Improving Visual-Semantic Embeddings with Hard Negatives
• Predicting Visual Features from Text for Image and Video Caption Retrieval
• Deep Hashing with Category Mask for Fast Video Retrieval
• Content-based Video Relevance Prediction Challenge: Data, Protocol, and Baseline
• FusedLSTM: Fusing frame-level and video-level features for Content-based Video Relevance Prediction

Cited by in corpus (4)

• Fine-Grained Fashion Similarity Prediction by Attribute-Specific Embedding Learning
• A Feature-space Multimodal Data Augmentation Technique for Text-video Retrieval
• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• What Matters for Ad-hoc Video Search? A Large-scale Evaluation on TRECVID