Learning to Collocate Neural Modules for Image Captioning

Abstract

We do not speak word by word from scratch; our brain quickly structures a pattern like \textsc{sth do sth at someplace} and then fill in the detailed descriptions. To render existing encoder-decoder image captioners such human-like reasoning, we propose a novel framework: learning to Collocate Neural Modules (CNM), to generate the `inner pattern' connecting visual encoder and language decoder. Unlike the widely-used neural module networks in visual Q\&A, where the language (ie, question) is fully observable, CNM for captioning is more challenging as the language is being generated and thus is partially observable. To this end, we make the following technical contributions for CNM training: 1) compact module design --- one for function words and three for visual content words (eg, noun, adjective, and verb), 2) soft module fusion and multi-step module execution, robustifying the visual reasoning in partial observation, 3) a linguistic loss for module controller being faithful to part-of-speech collocations (eg, adjective is before noun). Extensive experiments on the challenging MS-COCO image captioning benchmark validate the effectiveness of our CNM image captioner. In particular, CNM achieves a new state-of-the-art 127.9 CIDEr-D on Karpathy split and a single-model 126.0 c40 on the official server. CNM is also robust to few training samples, eg, by training only one sentence per image, CNM can halve the performance loss compared to a strong baseline.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Show, Attend and Tell: Neural Image Caption Generation with Visual Attention
• Categorical Reparameterization with Gumbel-Softmax
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Context-Aware Visual Policy Network for Sequence-Level Image Captioning
• Stack-Captioning: Coarse-to-Fine Learning for Image Captioning
• Learning to Assemble Neural Module Tree Networks for Visual Grounding
• Auto-Encoding Scene Graphs for Image Captioning

Cited by in corpus (2)

• Counterfactual VQA: A Cause-Effect Look at Language Bias
• MUTATT: Visual-Textual Mutual Guidance for Referring Expression Comprehension