SalGAN: Visual Saliency Prediction with Generative Adversarial Networks

Abstract

We introduce SalGAN, a deep convolutional neural network for visual saliency prediction trained with adversarial examples. The first stage of the network consists of a generator model whose weights are learned by back-propagation computed from a binary cross entropy (BCE) loss over downsampled versions of the saliency maps. The resulting prediction is processed by a discriminator network trained to solve a binary classification task between the saliency maps generated by the generative stage and the ground truth ones. Our experiments show how adversarial training allows reaching state-of-the-art performance across different metrics when combined with a widely-used loss function like BCE. Our results can be reproduced with the source code and trained models available at https://imatge-upc.github.io/saliency-salgan-2017/.

arXiv commentSubmitted for review to Computer Vision and Image Understanding (CVIU)

Cited by in corpus (51)

• RGB-D Salient Object Detection: A Survey
• Generative Adversarial Networks in Computer Vision: A Survey and Taxonomy
• Predicting Head Movement in Panoramic Video: A Deep Reinforcement Learning Approach
• A Review on Generative Adversarial Networks: Algorithms, Theory, and Applications
• Contextual Encoder-Decoder Network for Visual Saliency Prediction
• Predicting Video Saliency with Object-to-Motion CNN and Two-layer Convolutional LSTM
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• SaltiNet: Scan-path Prediction on 360 Degree Images using Saliency Volumes
• ConnNet: A Long-Range Relation-Aware Pixel-Connectivity Network for Salient Segmentation
• Simple vs complex temporal recurrences for video saliency prediction
• Systematic Review of Extended Reality for Smart Built Environments Lighting Design Simulations
• Saliency Prediction in the Deep Learning Era: Successes, Limitations, and Future Challenges
• Rethinking Object Saliency Ranking: A Novel Whole-flow Processing Paradigm
• TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection
• Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks
• Attention-aware Multi-stroke Style Transfer
• What Catches the Eye? Visualizing and Understanding Deep Saliency Models
• A Dilated Inception Network for Visual Saliency Prediction
• Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
• ATSal: An Attention Based Architecture for Saliency Prediction in 360 Videos
• ST-MTL: Spatio-Temporal Multitask Learning Model to Predict Scanpath While Tracking Instruments in Robotic Surgery
• DSAL-GAN: Denoising based Saliency Prediction with Generative Adversarial Networks
• State-of-the-art in 360° Video/Image Processing: Perception, Assessment and Compression
• Adversarial Attacks against Deep Saliency Models
• Instance-Level Relative Saliency Ranking with Graph Reasoning
• Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network
• RecSal : Deep Recursive Supervision for Visual Saliency Prediction
• SalSum: Saliency-based Video Summarization using Generative Adversarial Networks
• Saliency-based Multiple Region of Interest Detection from a Single 360° image
• Toward Better Understanding of Saliency Prediction in Augmented 360 Degree Videos
• Cube Padding for Weakly-Supervised Saliency Prediction in 360° Videos
• Saliency Prediction on Omnidirectional Images with Generative Adversarial Imitation Learning
• Believe It or Not, We Know What You Are Looking at!
• SalGaze: Personalizing Gaze Estimation Using Visual Saliency
• Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos
• Disentangling top-down vs. bottom-up and low-level vs. high-level influences on eye movements over time
• A New Ensemble Adversarial Attack Powered by Long-term Gradient Memories
• Light-weighted Saliency Detection with Distinctively Lower Memory Cost and Model Size
• Coupled Generative Adversarial Network for Continuous Fine-grained Action Segmentation
• $n$-Reference Transfer Learning for Saliency Prediction
• Quality-Aware Multimodal Saliency Detection via Deep Reinforcement Learning
• Temporal Saliency Adaptation in Egocentric Videos
• Saliency Benchmarking Made Easy: Separating Models, Maps and Metrics
• Learning Where to Look While Tracking Instruments in Robot-assisted Surgery
• DAF-NET: a saliency based weakly supervised method of dual attention fusion for fine-grained image classification
• \emph{cm}SalGAN: RGB-D Salient Object Detection with Cross-View Generative Adversarial Networks
• Grounded Human-Object Interaction Hotspots from Video (Extended Abstract)
• Salient Bundle Adjustment for Visual SLAM
• Implicit Saliency in Deep Neural Networks
• Unsupervised Meta-learning of Figure-Ground Segmentation via Imitating Visual Effects
• EEG-Driven Image Reconstruction with Saliency-Guided Diffusion Models