CPTR: Full Transformer Network for Image Captioning
arXiv:2101.10804
Abstract
In this paper, we consider the image captioning task from a new sequence-to-sequence prediction perspective and propose CaPtion TransformeR (CPTR) which takes the sequentialized raw images as the input to Transformer. Compared to the "CNN+Transformer" design paradigm, our model can model global context at every encoder layer from the beginning and is totally convolution-free. Extensive experiments demonstrate the effectiveness of the proposed model and we surpass the conventional "CNN+Transformer" methods on the MSCOCO dataset. Besides, we provide detailed visualizations of the self-attention between patches in the encoder and the "words-to-patches" attention in the decoder thanks to the full Transformer architecture.
References in corpus (3)
Cited by in corpus (15)
- Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos
- Deep Learning Approaches on Image Captioning: A Review
- Adjacent-Level Feature Cross-Fusion With 3-D CNN for Remote Sensing Image Change Detection
- Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks
- Multiscale Vision Transformers
- Audio Captioning Transformer
- SYMBA: Symbolic Computation of Squared Amplitudes in High Energy Physics with Machine Learning
- Recent Advances and Trends in Multimodal Deep Learning: A Review
- Radiology Report Generation Using Transformers Conditioned with Non-imaging Data
- ClipCap: CLIP Prefix for Image Captioning
- Describing image focused in cognitive and visual details for visually impaired people: An approach to generating inclusive paragraphs
- Automatic Generation of Product-Image Sequence in E-commerce
- ImageNet-21K Pretraining for the Masses
- Partially-Supervised Novel Object Captioning Leveraging Context from Paired Data
- Exploration into Translation-Equivariant Image Quantization