Seeing with Humans: Gaze-Assisted Neural Image Captioning

Abstract

Gaze reflects how humans process visual scenes and is therefore increasingly used in computer vision systems. Previous works demonstrated the potential of gaze for object-centric tasks, such as object localization and recognition, but it remains unclear if gaze can also be beneficial for scene-centric tasks, such as image captioning. We present a new perspective on gaze-assisted image captioning by studying the interplay between human gaze and the attention mechanism of deep neural networks. Using a public large-scale gaze dataset, we first assess the relationship between state-of-the-art object and scene recognition models, bottom-up visual saliency, and human gaze. We then propose a novel split attention model for image captioning. Our model integrates human gaze information into an attention-based long short-term memory architecture, and allows the algorithm to allocate attention selectively to both fixated and non-fixated image regions. Through evaluation on the COCO/SALICON datasets we show that our method improves image captioning performance and that gaze can complement machine attention for semantic scene understanding tasks.

References in corpus (12)

• Adam: A Method for Stochastic Optimization
• Microsoft COCO Captions: Data Collection and Evaluation Server
• DRAW: A Recurrent Neural Network For Image Generation
• TurkerGaze: Crowdsourcing Saliency with Webcam based Eye Tracking
• Image Captioning with Semantic Attention
• Places205-VGGNet Models for Scene Recognition
• Language Models for Image Captioning: The Quirks and What Works
• Learning like a Child: Fast Novel Visual Concept Learning from Sentence Descriptions of Images
• CIDEr: Consensus-based Image Description Evaluation
• On Learning Where To Look
• Crowdsourcing Gaze Data Collection
• GazeDPM: Early Integration of Gaze Information in Deformable Part Models

Cited by in corpus (7)

• Improving Natural Language Processing Tasks with Human Gaze-Guided Neural Attention
• Multimodal Integration of Human-Like Attention in Visual Question Answering
• VQA-MHUG: A Gaze Dataset to Study Multimodal Neural Attention in Visual Question Answering
• A Comprehensive Survey of Deep Learning for Image Captioning
• Paying More Attention to Saliency: Image Captioning with Saliency and Context Attention
• Gaze Embeddings for Zero-Shot Image Classification
• Boosted Attention: Leveraging Human Attention for Image Captioning