187 citations · 475 across the 25 of their papers we have counts for
32 papers
LMCap: Few-shot Multilingual Image Captioning by Retrieval Augmented Language Model Prompting
Rita Ramos, Bruno Martins, Desmond Elliott
Multilingual image captioning has recently been tackled by training with large-scale machine translated data, which is an expensive, noisy, and time-consuming process. Without requ…
The Role of Data Curation in Image Captioning
Wenyan Li, Jonas F. Lotz, Chen Qiu +1
Image captioning models are typically trained by treating all samples equally, neglecting to account for mismatched or otherwise difficult data points. In contrast, recent work has…
Retrieval-augmented Image Captioning
Rita Ramos, Desmond Elliott, Bruno Martins
Inspired by retrieval-augmented language generation and pretrained Vision and Language (V&L) encoders, we present a new approach to image captioning that generates sentences given…
Multilingual Multimodal Learning with Machine Translated Text
Chen Qiu, Dan Oneata, Emanuele Bugliarello +2
Most vision-and-language pretraining research focuses on English tasks. However, the creation of multilingual multimodal evaluation datasets (e.g. Multi30K, xGQA, XVNLI, and MaRVL)…
An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification
Ilias Chalkidis, Xiang Dai, Manos Fergadiotis +2
Non-hierarchical sparse attention Transformer-based models, such as Longformer and Big Bird, are popular approaches to working with long documents. There are clear benefits to thes…
SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentation
Rita Ramos, Bruno Martins, Desmond Elliott +1
Recent advances in image captioning have focused on scaling the data and model size, substantially increasing the cost of pre-training and finetuning. As an alternative to large mo…
Language Modelling with Pixels
Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello +3
Language models are defined over a finite set of inputs, which creates a vocabulary bottleneck when we attempt to scale the number of supported languages. Tackling this bottleneck…
Revisiting Transformer-based Models for Long Document Classification
Xiang Dai, Ilias Chalkidis, Sune Darkner +1
The recent literature in text classification is biased towards short text sequences (e.g., sentences or paragraphs). In real-world applications, multi-page multi-paragraph document…
IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
Emanuele Bugliarello, Fangyu Liu, Jonas Pfeiffer +4
Reliable evaluation benchmarks designed for replicability and comprehensiveness have driven progress in machine learning. Due to the lack of a multilingual benchmark, however, visi…
Visually Grounded Reasoning across Languages and Cultures
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti +3
The design of widespread vision-and-language datasets and pre-trained encoders directly adopts, or draws inspiration from, the concepts and images of ImageNet. While one can hardly…
MDAPT: Multilingual Domain Adaptive Pretraining in a Single Model
Rasmus Kær Jørgensen, Mareike Hartmann, Xiang Dai +1
Domain adaptive pretraining, i.e. the continued unsupervised pretraining of a language model on domain-specific text, improves the modelling of text for downstream tasks within the…
Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
Stella Frank, Emanuele Bugliarello, Desmond Elliott
Pretrained vision-and-language BERTs aim to learn representations that combine information from both modalities. We propose a diagnostic method based on cross-modal input ablation…
The Role of Syntactic Planning in Compositional Image Captioning
Emanuele Bugliarello, Desmond Elliott
Image captioning has focused on generalizing to images drawn from the same distribution as the training set, and not to the more challenging problem of generalizing to different di…
Multimodal Pretraining Unmasked: A Meta-Analysis and a Unified Framework of Vision-and-Language BERTs
Emanuele Bugliarello, Ryan Cotterell, Naoaki Okazaki +1
Large-scale pretraining and task-specific fine-tuning is now the standard methodology for many tasks in computer vision and natural language processing. Recently, a multitude of me…
Multimodal Speech Recognition with Unstructured Audio Masking
Tejas Srinivasan, Ramon Sanabria, Florian Metze +1
Visual context has been shown to be useful for automatic speech recognition (ASR) systems when the speech signal is noisy or corrupted. Previous work, however, has only demonstrate…
Fine-Grained Grounding for Multimodal Speech Recognition
Tejas Srinivasan, Ramon Sanabria, Florian Metze +1
Multimodal automatic speech recognition systems integrate information from images to improve speech recognition quality, by grounding the speech in the visual context. While visual…
CompGuessWhat?!: A Multi-task Evaluation Framework for Grounded Language Learning
Alessandro Suglia, Ioannis Konstas, Andrea Vanzo +4
Approaches to Grounded Language Learning typically focus on a single task-based final performance measure that may not depend on desirable properties of the learned hidden represen…
The Sensitivity of Language Models and Humans to Winograd Schema Perturbations
Mostafa Abdou, Vinit Ravishankar, Maria Barrett +3
Large-scale pretrained language models are the major driving force behind recent improvements in performance on the Winograd Schema Challenge, a widely employed test of common sens…
Multimodal Machine Translation through Visuals and Speech
Umut Sulubacak, Ozan Caglayan, Stig-Arne Grönroos +4
Multimodal machine translation involves drawing information from more than one modality, based on the assumption that the additional modalities will contain useful alternative view…
Bootstrapping Disjoint Datasets for Multilingual Multimodal Representation Learning
Ákos Kádár, Grzegorz Chrupała, Afra Alishahi +1
Recent work has highlighted the advantage of jointly learning grounded sentence representations from multiple languages. However, the data used in these studies has been limited to…
Compositional Generalization in Image Captioning
Mitja Nikolaus, Mostafa Abdou, Matthew Lamm +2
Image captioning models are usually evaluated on their ability to describe a held-out set of images, not on their ability to generalize to unseen concepts. We study the problem of…
Cross-lingual Visual Verb Sense Disambiguation
Spandana Gella, Desmond Elliott, Frank Keller
Recent work has shown that visual context improves cross-lingual sense disambiguation for nouns. We extend this line of work to the more challenging task of cross-lingual verb sens…
How2: A Large-scale Dataset for Multimodal Language Understanding
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar +4
In this paper, we introduce How2, a multimodal collection of instructional videos with English subtitles and crowdsourced Portuguese translations. We also present integrated sequen…
Lessons learned in multilingual grounded language learning
Ákos Kádár, Desmond Elliott, Marc-Alexandre Côté +2
Recent work has shown how to learn better visual-semantic embeddings by leveraging image descriptions in more than one language. Here, we investigate in detail which conditions aff…