Deconfounded Image Captioning: A Causal Retrospect

Abstract

Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.

References in corpus (3)

• Shortcut Learning in Deep Neural Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Context-Aware Visual Policy Network for Fine-Grained Image Captioning

Cited by in corpus (7)

• Cross-Modal Causal Intervention for Medical Report Generation
• Embedded Heterogeneous Attention Transformer for Cross-lingual Image Captioning
• Towards Deconfounded Image-Text Matching with Causal Inference
• Mitigating Prior Errors in Causal Structure Learning: A Resilient Approach via Bayesian Networks
• Structure Causal Models and LLMs Integration in Medical Visual Question Answering
• Visual Superordinate Abstraction for Robust Concept Learning
• Backdoor Adjustment of Confounding by Provenance for Robust Text Classification of Multi-institutional Clinical Notes