Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach

Abstract

Panoramic video provides immersive and interactive experience by enabling humans to control the field of view (FoV) through head movement (HM). Thus, HM plays a key role in modeling human attention on panoramic video. This paper establishes a database collecting subjects' HM in panoramic video sequences. From this database, we find that the HM data are highly consistent across subjects. Furthermore, we find that deep reinforcement learning (DRL) can be applied to predict HM positions, via maximizing the reward of imitating human HM scanpaths through the agent's actions. Based on our findings, we propose a DRL-based HM prediction (DHP) approach with offline and online versions, called offline-DHP and online-DHP. In offline-DHP, multiple DRL workflows are run to determine potential HM positions at each panoramic frame. Then, a heat map of the potential HM positions, named the HM map, is generated as the output of offline-DHP. In online-DHP, the next HM position of one subject is estimated given the currently observed HM position, which is achieved by developing a DRL algorithm upon the learned offline-DHP model. Finally, the experiments validate that our approach is effective in both offline and online prediction of HM positions for panoramic video, and that the learned offline-DHP model can improve the performance of online-DHP.

15 pages, 10 figures, published on TPAMI 2018

References in corpus (5)

• Recurrent Models of Visual Attention
• Video Salient Object Detection via Fully Convolutional Networks
• SalGAN: Visual Saliency Prediction with Generative Adversarial Networks
• Reinforcement Learning with Unsupervised Auxiliary Tasks
• SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes

Cited by in corpus (29)

• Assessing Visual Quality of Omnidirectional Videos
• Multimodality in VR: A survey
• Taming the latency in multi-user VR 360${}^{\circ }$: A QoE-aware deep learning-aided multicast framework
• ScanGAN360: A Generative Model of Realistic Scanpaths for 360${}^{\circ }$ Images
• MO-MIX: Multi-Objective Multi-Agent Cooperative Decision-Making With Deep Reinforcement Learning
• Spherical clustering of users navigating 360° content
• Systematic Review of Extended Reality for Smart Built Environments Lighting Design Simulations
• MADRL-Based Rate Adaptation for 360° Video Streaming with Multi-Viewpoint Prediction
• Understanding and Predicting the Memorability of Outdoor Natural Scenes
• Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model
• Tile Classification Based Viewport Prediction with Multi-modal Fusion Transformer
• Immersive Multimedia Communication: State-of-the-Art on eXtended Reality Streaming
• Blind Omnidirectional Image Quality Assessment with Viewport Oriented Graph Convolutional Networks
• Panonut360: A Head and Eye Tracking Dataset for Panoramic Video
• ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos
• State-of-the-art in 360° Video/Image Processing: Perception, Assessment and Compression
• ASOD60K: An Audio-Induced Salient Object Detection Dataset for Panoramic Videos
• Stereoscopic Omnidirectional Image Quality Assessment Based on Predictive Coding Theory
• Revisiting Deep Architectures for Head Motion Prediction in 360° Videos
• Generating Realistic Synthetic Head Rotation Data for Extended Reality using Deep Learning
• Diversity-Driven Extensible Hierarchical Reinforcement Learning
• Prediction, Communication, and Computing Duration Optimization for VR Video Streaming
• Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning
• QoE Driven VR 360 Video Massive MIMO Transmission
• Probabilistic Tile Visibility-Based Server-Side Rate Adaptation for Adaptive 360-Degree Video Streaming
• A Fixation-based 360° Benchmark Dataset for Salient Object Detection
• SHD360: A Benchmark Dataset for Salient Human Detection in 360° Videos
• Distortion-adaptive Salient Object Detection in 360${}^{\circ }$ Omnidirectional Images
• Virtual Reality: A Survey of Enabling Technologies and its Applications in IoT