Spatiotemporal Knowledge Distillation for Efficient Estimation of Aerial Video Saliency

Abstract

The performance of video saliency estimation techniques has achieved significant advances along with the rapid development of Convolutional Neural Networks (CNNs). However, devices like cameras and drones may have limited computational capability and storage space so that the direct deployment of complex deep saliency models becomes infeasible. To address this problem, this paper proposes a dynamic saliency estimation approach for aerial videos via spatiotemporal knowledge distillation. In this approach, five components are involved, including two teachers, two students and the desired spatiotemporal model. The knowledge of spatial and temporal saliency is first separately transferred from the two complex and redundant teachers to their simple and compact students, and the input scenes are also degraded from high-resolution to low-resolution to remove the probable data redundancy so as to greatly speed up the feature extraction process. After that, the desired spatiotemporal model is further trained by distilling and encoding the spatial and temporal saliency knowledge of two students into a unified network. In this manner, the inter-model redundancy can be further removed for the effective estimation of dynamic saliency on aerial videos. Experimental results show that the proposed approach outperforms ten state-of-the-art models in estimating visual saliency on aerial videos, while its speed reaches up to 28,738 FPS on the GPU platform.

References in corpus (10)

• YouTube-8M: A Large-Scale Video Classification Benchmark
• Video Salient Object Detection via Fully Convolutional Networks
• Deep Visual Attention Prediction
• Visual Saliency Based on Scale-Space Analysis in the Frequency Domain
• DeepSaliency: Multi-Task Deep Neural Network Model for Salient Object Detection
• Adversarial Spatio-Temporal Learning for Video Deblurring
• DeepGaze II: Reading fixations from deep features trained on object recognition
• Deep Gaze I: Boosting Saliency Prediction with Feature Maps Trained on ImageNet
• Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM
• Exploiting inter-image similarity and ensemble of extreme learners for fixation prediction using deep features

Cited by in corpus (2)

• Benchmarking Micro-action Recognition: Dataset, Methods, and Applications
• ResKD: Residual-Guided Knowledge Distillation