Overcoming Statistical Shortcuts for Open-ended Visual Counting

Abstract

Machine learning models tend to over-rely on statistical shortcuts. These spurious correlations between parts of the input and the output labels does not hold in real-world settings. We target this issue on the recent open-ended visual counting task which is well suited to study statistical shortcuts. We aim to develop models that learn a proper mechanism of counting regardless of the output label. First, we propose the Modifying Count Distribution (MCD) protocol, which penalizes models that over-rely on statistical shortcuts. It is based on pairs of training and testing sets that do not follow the same count label distribution such as the odd-even sets. Intuitively, models that have learned a proper mechanism of counting on odd numbers should perform well on even numbers. Secondly, we introduce the Spatial Counting Network (SCN), which is dedicated to visual analysis and counting based on natural language questions. Our model selects relevant image regions, scores them with fusion and self-attention mechanisms, and provides a final counting score. We apply our protocol on the recent dataset, TallyQA, and show superior performances compared to state-of-the-art models. We also demonstrate the ability of our model to select the correct instances to count in the image. Code and datasets are available: https://github.com/cdancette/spatial-counting-network

17 pages, 8 figures

References in corpus (11)

• Efficient Estimation of Word Representations in Vector Space
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Shortcut Learning in Deep Neural Networks
• Skip-Thought Vectors
• A Survey of Recent Advances in CNN-based Single Image Crowd Counting and Density Estimation
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Hadamard Product for Low-rank Bilinear Pooling
• Self-Critical Reasoning for Robust Visual Question Answering
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law
• Interpretable Counting for Visual Question Answering