Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos

Abstract

Automatic saliency prediction in 360° videos is critical for viewpoint guidance applications (e.g., Facebook 360 Guide). We propose a spatial-temporal network which is (1) weakly-supervised trained and (2) tailor-made for 360° viewing sphere. Note that most existing methods are less scalable since they rely on annotated saliency map for training. Most importantly, they convert 360° sphere to 2D images (e.g., a single equirectangular image or multiple separate Normal Field-of-View (NFoV) images) which introduces distortion and image boundaries. In contrast, we propose a simple and effective Cube Padding (CP) technique as follows. Firstly, we render the 360° view on six faces of a cube using perspective projection. Thus, it introduces very little distortion. Then, we concatenate all six faces while utilizing the connectivity between faces on the cube for image padding (i.e., Cube Padding) in convolution, pooling, convolutional LSTM layers. In this way, CP introduces no image boundary while being applicable to almost all Convolutional Neural Network (CNN) structures. To evaluate our method, we propose Wild-360, a new 360° video saliency dataset, containing challenging videos with saliency heatmap annotations. In experiments, our method outperforms baseline methods in both speed and quality.

arXiv commentCVPR 2018

References in corpus (13)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
• Deep Residual Learning for Image Recognition
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue
• Learning Deep Features for Discriminative Localization
• Delving Deeper into Convolutional Networks for Learning Video Representations
• Learning Spherical Convolution for Fast Features from 360° Imagery
• Shallow and Deep Convolutional Networks for Saliency Prediction
• Learning Video Object Segmentation with Visual Memory
• Semantic-driven Generation of Hyperlapse from $360^{\circ }$ Video
• A Deep Ranking Model for Spatio-Temporal Highlight Detection from a 360 Video
• Viraliency: Pooling Local Virality

Cited by in corpus (2)

• Revisiting Optical Flow Estimation in 360 Videos
• Egok360: A 360 Egocentric Kinetic Human Activity Video Dataset