111 citations · 257 across the 26 of their papers we have counts for
29 papers
LAION-BVD: A 10-Million-Hour Open Video Dataset for Multimodal Pre-training
Andreas Hochlehnert, Marianna Nezhurina, Mehdi Cherti +9
We present LAION-BVD, a large-scale open video dataset for multimodal learning, which contains 1.3B platform-specific video URLs collected from CommonCrawl. From these, we download…
DF-MoE: Generalizable Deepfake Detection via Multimodal Sparse Mixture-of-Experts
Vlad Hondru, Florinel Alin Croitoru, Iuliana Georgescu +2
Audio-visual deepfake detection is an actively studied topic, where one of the main challenges is to develop detectors able to generalize across deepfake generation methods. We con…
Back into Plato's Cave: Examining Cross-modal Representational Convergence at Scale
A. Sophia Koepke, Daniil Zverev, Shiry Ginosar +1
The Platonic Representation Hypothesis posits that neural networks trained on different modalities (e.g., text and images) converge toward a shared representation of reality. If tr…
It's Never Too Late: Noise Optimization for Collapse Recovery in Trained Diffusion Models
Anne Harrington, A. Sophia Koepke, Shyamgopal Karthik +2
Contemporary text-to-image models exhibit a surprising degree of mode collapse, as can be seen when sampling several images given the same text prompt. Previous work has attempted…
On the Dangers of Bootstrapping Generation for Continual Learning and Beyond
Daniil Zverev, A. Sophia Koepke, Joao F. Henriques
The use of synthetically generated data for training models is becoming a common practice. While generated data can augment the training data, repeated training on synthetic data r…
VGGSounder: Audio-Visual Evaluations for Foundation Models
Daniil Zverev, Thaddäus Wiedemer, Ameya Prabhu +3
The emergence of audio-visual foundation models underscores the importance of reliably assessing their multi-modal understanding. The VGGSound dataset is commonly used as a benchma…
Dissecting Temporal Understanding in Text-to-Audio Retrieval
Andreea-Maria Oncescu, João F. Henriques, A. Sophia Koepke
Recent advancements in machine learning have fueled research on multimodal tasks, such as for instance text-to-video and text-to-audio retrieval. These tasks require models to unde…
Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
David Kurzendörfer, Otniel-Bogdan Mercea, A. Sophia Koepke +1
Audio-visual zero-shot learning methods commonly build on features extracted from pre-trained models, e.g. video or audio classification models. However, existing benchmarks predat…
A SOUND APPROACH: Using Large Language Models to generate audio descriptions for egocentric text-audio retrieval
Andreea-Maria Oncescu, João F. Henriques, Andrew Zisserman +2
Video databases from the internet are a valuable source of text-audio retrieval datasets. However, given that sound and vision streams represent different "views" of the data, trea…
Zero-shot audio captioning with audio-language model guidance and audio context keywords
Leonard Salewski, Stefan Fauth, A. Sophia Koepke +1
Zero-shot audio captioning aims at automatically generating descriptive textual captions for audio content without prior training for this task. Different from speech recognition w…
Zero-shot Translation of Attention Patterns in VQA Models to Natural Language
Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch +1
Converting a model's internals to text can yield human-understandable insights about the model. Inspired by the recent success of training-free approaches for image captioning, we…
Fantastic Gains and Where to Find Them: On the Existence and Prospect of General Knowledge Transfer between Any Pretrained Model
Karsten Roth, Lukas Thede, Almut Sophia Koepke +3
Training deep networks requires various design decisions regarding for instance their architecture, data augmentation, or optimization. In this work, we find these training variati…
Video-adverb retrieval with compositional adverb-action embeddings
Thomas Hummel, Otniel-Bogdan Mercea, A. Sophia Koepke +1
Retrieving adverbs that describe an action in a video poses a crucial step towards fine-grained video understanding. We propose a framework for video-to-adverb retrieval (and vice…
Text-to-feature diffusion for audio-visual few-shot learning
Otniel-Bogdan Mercea, Thomas Hummel, A. Sophia Koepke +1
Training deep learning models for video classification from audio-visual data commonly requires immense amounts of labeled training data collected via a costly process. A challengi…
Image-free Classifier Injection for Zero-Shot Classification
Anders Christensen, Massimiliano Mancini, A. Sophia Koepke +2
Zero-shot learning models achieve remarkable results on image classification for samples from classes that were not seen during training. However, such models must be trained from…
Addressing caveats of neural persistence with deep graph persistence
Leander Girrbach, Anders Christensen, Ole Winther +2
Neural Persistence is a prominent measure for quantifying neural network complexity, proposed in the emerging field of topological data analysis in deep learning. In this work, how…
Waffling around for Performance: Visual Classification with Random Words and Broad Concepts
Karsten Roth, Jae Myung Kim, A. Sophia Koepke +3
The visual classification performance of vision-language models such as CLIP has been shown to benefit from additional semantic knowledge from large language models (LLMs) such as…
Exposing and Mitigating Spurious Correlations for Cross-Modal Retrieval
Jae Myung Kim, A. Sophia Koepke, Cordelia Schmid +1
Cross-modal retrieval methods are the preferred tool to search databases for the text that best matches a query image and vice versa. However, image-text retrieval models commonly…
PlanT: Explainable Planning Transformers via Object-Level Representations
Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea +3
Planning an optimal route in a complex environment requires efficient reasoning about the surrounding scene. While human drivers prioritize important objects and ignore details not…
Temporal and cross-modal attention for audio-visual zero-shot learning
Otniel-Bogdan Mercea, Thomas Hummel, A. Sophia Koepke +1
Audio-visual generalised zero-shot learning for video classification requires understanding the relations between the audio and visual information in order to be able to recognise…
CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations
Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch +1
Providing explanations in the context of Visual Question Answering (VQA) presents a fundamental problem in machine learning. To obtain detailed insights into the process of generat…
Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language
Otniel-Bogdan Mercea, Lukas Riesch, A. Sophia Koepke +1
Learning to classify video data from classes not included in the training data, i.e. video-based zero-shot learning, is challenging. We conjecture that the natural alignment betwee…
Audio Retrieval with Natural Language Queries: A Benchmark Study
A. Sophia Koepke, Andreea-Maria Oncescu, João F. Henriques +2
The objectives of this work are cross-modal text-audio and audio-text retrieval, in which the goal is to retrieve the audio content from a pool of candidates that best matches a gi…
Where and When: Space-Time Attention for Audio-Visual Explanations
Yanbei Chen, Thomas Hummel, A. Sophia Koepke +1
Explaining the decision of a multi-modal decision-maker requires to determine the evidence from both modalities. Recent advances in XAI provide explanations for models trained on s…