MoVie: Revisiting Modulated Convolutions for Visual Counting and Beyond

Abstract

This paper focuses on visual counting, which aims to predict the number of occurrences given a natural image and a query (e.g. a question or a category). Unlike most prior works that use explicit, symbolic models which can be computationally expensive and limited in generalization, we propose a simple and effective alternative by revisiting modulated convolutions that fuse the query and the image locally. Following the design of residual bottleneck, we call our method MoVie, short for Modulated conVolutional bottlenecks. Notably, MoVie reasons implicitly and holistically and only needs a single forward-pass during inference. Nevertheless, MoVie showcases strong performance for counting: 1) advancing the state-of-the-art on counting-specific VQA tasks while being more efficient; 2) outperforming prior-art on difficult benchmarks like COCO for common object counting; 3) helped us secure the first place of 2020 VQA challenge when integrated as a module for 'number' related questions in generic VQA models. Finally, we show evidence that modulated convolutions such as MoVie can serve as a general mechanism for reasoning tasks beyond counting.

References in corpus (17)

• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Improving neural networks by preventing co-adaptation of feature detectors
• VQA: Visual Question Answering
• A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation
• Multimodal Compact Bilinear Pooling for Visual Question Answering and Visual Grounding
• Modulating early visual processing by language
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Learning to Count Objects in Natural Images for Visual Question Answering
• Pythia v0.1: the Winning Entry to the VQA Challenge 2018
• Compositional Attention Networks for Machine Reasoning
• Learning to Reason: End-to-End Module Networks for Visual Question Answering
• Multi-modal Factorized Bilinear Pooling with Co-Attention Learning for Visual Question Answering
• Interpretable Counting for Visual Question Answering
• What Makes Training Multi-Modal Classification Networks Hard?
• Object Counting and Instance Segmentation with Image-level Supervision
• In Defense of Grid Features for Visual Question Answering
• The Visual QA Devil in the Details: The Impact of Early Fusion and Batch Norm on CLEVR

Cited by in corpus (4)

• ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision
• Less is More: ClipBERT for Video-and-Language Learning via Sparse Sampling
• Achieving Human Parity on Visual Question Answering
• Cross-Modal Retrieval Augmentation for Multi-Modal Classification