Abstract
Dataset bias in vision-language tasks is becoming one of the main problems which hinders the progress of our community. Existing solutions lack a principled analysis about why modern image captioners easily collapse into dataset bias. In this paper, we present a novel perspective: Deconfounded Image Captioning (DIC), to find out the answer of this question, then retrospect modern neural image captioners, and finally propose a DIC framework: DICv1.0 to alleviate the negative effects brought by dataset bias. DIC is based on causal inference, whose two principles: the backdoor and front-door adjustments, help us review previous studies and design new effective models. In particular, we showcase that DICv1.0 can strengthen two prevailing captioning models and can achieve a single-model 131.1 CIDEr-D and 128.4 c40 CIDEr-D on Karpathy split and online split of the challenging MS COCO dataset, respectively. Interestingly, DICv1.0 is a natural derivation from our causal retrospect, which opens promising directions for image captioning.
| Original language | English |
|---|---|
| Pages (from-to) | 12996-13010 |
| Number of pages | 14 |
| Journal | IEEE Transactions on Pattern Analysis and Machine Intelligence |
| Volume | 45 |
| Issue number | 11 |
| DOIs | |
| Publication status | Published - 1 Nov 2023 |
Keywords
- Causality
- Correlation
- Deconfounding
- Image Captioning
- Magnetic heads
- Shape
- Task analysis
- the Backdoor Adjustment
- the Front-Door Adjustment
- Toy manufacturing industry
- Training
- Visualization
Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver