75.5k citations · 100.8k across the 160 of their papers we have counts for
198 papers
Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation
Akam Rahimi, Triantafyllos Afouras, Andrew Zisserman
The goal of this paper is speech separation and enhancement in multi-speaker and noisy environments using a combination of different modalities. Previous works have shown good perf…
Predicting Spine Geometry and Scoliosis from DXA Scans
Amir Jamaludin, Timor Kadir, Emma Clark +1
Our objective in this paper is to estimate spine curvature in DXA scans. To this end we first train a neural network to predict the middle spine curve in the scan, and then use an…
The Change You Want to See (Now in 3D)
Ragav Sachdeva, Andrew Zisserman
The goal of this paper is to detect what has changed, if anything, between two "in the wild" images of the same 3D scene acquired from different camera positions and at different t…
The Making and Breaking of Camouflage
Hala Lamdouar, Weidi Xie, Andrew Zisserman
Not all camouflages are equally effective, as even a partially visible contour or a slight color difference can make the animal stand out and break its camouflage. In this paper, w…
OxfordVGG Submission to the EGO4D AV Transcription Challenge
Jaesung Huh, Max Bain, Andrew Zisserman
This report presents the technical details of our submission on the EGO4D Audio-Visual (AV) Automatic Speech Recognition Challenge 2023 from the OxfordVGG team. We present WhisperX…
Multi-Modal Classifiers for Open-Vocabulary Object Detection
Prannay Kaul, Weidi Xie, Andrew Zisserman
The goal of this paper is open-vocabulary object detection (OVOD) $\unicode{x2013}$ building a model that can detect objects beyond the set of categories seen at training, thus ena…
TAPIR: Tracking Any Point with per-frame Initialization and temporal Refinement
Carl Doersch, Yi Yang, Mel Vecerik +5
We present a novel model for Tracking Any Point (TAP) that effectively tracks any queried point on any physical surface throughout a video sequence. Our approach employs two stages…
Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion
Yash Bhalgat, Iro Laina, João F. Henriques +2
Instance segmentation in 3D is a challenging task due to the lack of large-scale annotated datasets. In this paper, we show that this task can be addressed effectively by leveragin…
Open-world Text-specified Object Counting
Niki Amini-Naieni, Kiana Amini-Naieni, Tengda Han +1
Our objective is open-world object counting in images, where the target object class is specified by a text description. To this end, we propose CounTX, a class-agnostic, single-st…
Perception Test: A Diagnostic Benchmark for Multimodal Video Models
Viorica Pătrăucean, Lucas Smaira, Ankush Gupta +21
We propose a novel multimodal video benchmark - the Perception Test - to evaluate the perception and reasoning skills of pre-trained multimodal models (e.g. Flamingo, SeViLA, or GP…
Verbs in Action: Improving verb understanding in video-language models
Liliane Momeni, Mathilde Caron, Arsha Nagrani +2
Understanding verbs is crucial to modelling how people and objects interact with each other and the environment through space and time. Recently, state-of-the-art video-language mo…
Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime
Rhydian Windsor, Amir Jamaludin, Timor Kadir +1
This paper explores training medical vision-language models (VLMs) -- where the visual and language inputs are embedded into a common space -- with a particular focus on scenarios…
AutoAD: Movie Description in Context
Tengda Han, Max Bain, Arsha Nagrani +3
The objective of this paper is an automatic Audio Description (AD) model that ingests movies and outputs AD in text form. Generating high-quality movie AD is challenging due to the…
Three ways to improve feature alignment for open vocabulary detection
Relja Arandjelović, Alex Andonian, Arthur Mensch +3
The core problem in zero-shot open vocabulary detection is how to align visual and text features, so that the detector performs well on unseen classes. Previous approaches train th…
VoxSRC 2022: The Fourth VoxCeleb Speaker Recognition Challenge
Jaesung Huh, Andrew Brown, Jee-weon Jung +4
This paper summarises the findings from the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22), which was held in conjunction with INTERSPEECH 2022. The goal of this challenge…
WhisperX: Time-Accurate Speech Transcription of Long-Form Audio
Max Bain, Jaesung Huh, Tengda Han +1
Large-scale, weakly-supervised speech recognition models, such as Whisper, have demonstrated impressive results on speech recognition across domains and languages. However, their a…
Zorro: the masked multimodal transformer
Adrià Recasens, Jason Lin, Joāo Carreira +8
Attention-based models are appealing for multimodal processing because inputs from multiple modalities can be concatenated and fed to a single backbone network - thus requiring ver…
Epic-Sounds: A Large-scale Dataset of Actions That Sound
Jaesung Huh, Jacob Chalk, Evangelos Kazakos +2
We introduce EPIC-SOUNDS, a large-scale dataset of audio annotations capturing temporal extents and class labels within the audio stream of the egocentric videos. We propose an ann…
Weakly-supervised Fingerspelling Recognition in British Sign Language Videos
K R Prajwal, Hannah Bull, Liliane Momeni +3
The goal of this work is to detect and recognize sequences of letters signed using fingerspelling in British Sign Language (BSL). Previous fingerspelling recognition methods have n…
A Light Touch Approach to Teaching Transformers Multi-view Geometry
Yash Bhalgat, Joao F. Henriques, Andrew Zisserman
Transformers are powerful visual learners, in large part due to their conspicuous lack of manually-specified priors. This flexibility can be problematic in tasks that involve multi…
TAP-Vid: A Benchmark for Tracking Any Point in a Video
Carl Doersch, Ankush Gupta, Larisa Markeeva +6
Generic motion understanding from video involves not only tracking objects, but also perceiving how their surfaces deform and move. This information is useful to make inferences ab…
End-to-end Tracking with a Multi-query Transformer
Bruno Korbar, Andrew Zisserman
Multiple-object tracking (MOT) is a challenging task that requires simultaneous reasoning about location, appearance, and identity of the objects in the scene over time. Our aim in…
A Tri-Layer Plugin to Improve Occluded Detection
Guanqi Zhan, Weidi Xie, Andrew Zisserman
Detecting occluded objects still remains a challenge for state-of-the-art object detectors. The objective of this work is to improve the detection for such objects, and thereby imp…
Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors
Vladimir Iashin, Weidi Xie, Esa Rahtu +1
The objective of this paper is audio-visual synchronisation of general videos 'in the wild'. For such videos, the events that may be harnessed for synchronisation cues may be spati…