266 citations · 1.8k across the 115 of their papers we have counts for
5 papers · 1 filter
One does not fit all! On the Complementarity of Vision Encoders for Vision and Language Tasks
Gregor Geigle, Chen Cecilia Liu, Jonas Pfeiffer +1
Current multimodal models, aimed at solving Vision and Language (V+L) tasks, predominantly repurpose Vision Encoders (VE) as feature extractors. While many VEs -- of different arch…
TxT: Crossmodal End-to-End Learning with Transformers
Jan-Martin O. Steitz, Jonas Pfeiffer, Iryna Gurevych +1
Reasoning over multiple modalities, e.g. in Visual Question Answering (VQA), requires an alignment of semantic concepts across domains. Despite the widespread success of end-to-end…
Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval
Gregor Geigle, Jonas Pfeiffer, Nils Reimers +2
Current state-of-the-art approaches to cross-modal retrieval process text and visual input jointly, relying on Transformer-based architectures with cross-attention mechanisms that…
Latent Normalizing Flows for Many-to-Many Cross-Domain Mappings
Shweta Mahajan, Iryna Gurevych, Stefan Roth
Learned joint representations of images and text form the backbone of several important cross-domain tasks such as image captioning. Prior work mostly maps both domains into a comm…
Joint Wasserstein Autoencoders for Aligning Multimodal Embeddings
Shweta Mahajan, Teresa Botschen, Iryna Gurevych +1
One of the key challenges in learning joint embeddings of multiple modalities, e.g. of images and text, is to ensure coherent cross-modal semantics that generalize across datasets.…