796 citations · 2.2k across the 23 of their papers we have counts for
31 papers
SAM 3: Segment Anything with Concepts
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu +35
We present Segment Anything Model (SAM) 3, a unified model that detects, segments, and tracks objects in images and videos based on concept prompts, which we define as either short…
Enrich and Detect: Video Temporal Grounding with Multimodal LLMs
Shraman Pramanick, Effrosyni Mavroudi, Yale Song +3
We introduce ED-VTG, a method for fine-grained video temporal grounding utilizing multi-modal large language models. Our approach harnesses the capabilities of multimodal LLMs to j…
PerceptionLM: Open-Access Data and Models for Detailed Visual Understanding
Jang Hyun Cho, Andrea Madotto, Effrosyni Mavroudi +26
Vision-language models are integral to computer vision research, yet many high-performing models remain closed-source, obscuring their data, design and training recipe. The researc…
Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation
Akam Rahimi, Triantafyllos Afouras, Andrew Zisserman
The goal of this paper is speech separation and enhancement in multi-speaker and noisy environments using a combination of different modalities. Previous works have shown good perf…
VoiceVector: Multimodal Enrolment Vectors for Speaker Separation
Akam Rahimi, Triantafyllos Afouras, Andrew Zisserman
We present a transformer-based architecture for voice separation of a target speaker from multiple other speakers and ambient noise. We achieve this by using two separate neural ne…
MusicFlow: Cascaded Flow Matching for Text Guided Music Generation
K R Prajwal, Bowen Shi, Matthew Lee +8
We introduce MusicFlow, a cascaded text-to-music generation model based on flow matching. Based on self-supervised representations to bridge between text descriptions and music aud…
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Kristen Grauman, Andrew Westbury, Lorenzo Torresani +98
We present Ego-Exo4D, a diverse, large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric…
Video-Mined Task Graphs for Keystep Recognition in Instructional Videos
Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras +1
Procedural activity understanding requires perceiving human actions in terms of a broader task, where multiple keysteps are performed in sequence across a long video to reach a fin…
Learning to Ground Instructional Articles in Videos through Narrations
Effrosyni Mavroudi, Triantafyllos Afouras, Lorenzo Torresani
In this paper we present an approach for localizing steps of procedural activities in narrated how-to videos. To deal with the scarcity of labeled data at scale, we source the step…
Scaling up sign spotting through sign language dictionaries
Gül Varol, Liliane Momeni, Samuel Albanie +2
The focus of this work is - given a video of an isolated sign, our task is to identify and it has been signed in a cont…
Audio-Visual Synchronisation in the wild
Honglie Chen, Weidi Xie, Triantafyllos Afouras +3
In this paper, we consider the problem of audio-visual synchronisation applied to videos `in-the-wild' (ie of general classes beyond speech). As a new task, we identify and curate…
BBC-Oxford British Sign Language Dataset
Samuel Albanie, Gül Varol, Liliane Momeni +8
In this work, we introduce the BBC-Oxford British Sign Language (BOBSL) dataset, a large-scale video collection of British Sign Language (BSL). BOBSL is an extended and publicly re…
Visual Keyword Spotting with Attention
K R Prajwal, Liliane Momeni, Triantafyllos Afouras +1
In this paper, we consider the task of spotting spoken keywords in silent video sequences -- also known as visual keyword spotting. To this end, we investigate Transformer-based mo…
Sub-word Level Lip Reading With Visual Attention
K R Prajwal, Triantafyllos Afouras, Andrew Zisserman
The goal of this paper is to learn strong lip reading models that can recognise speech in silent videos. Most prior works deal with the open-set visual speech recognition problem b…
Aligning Subtitles in Sign Language Videos
Hannah Bull, Triantafyllos Afouras, Gül Varol +3
The goal of this work is to temporally align asynchronous subtitles in sign language videos. In particular, we focus on sign-language interpreted TV broadcast data comprising (i) a…
Localizing Visual Sounds the Hard Way
Honglie Chen, Weidi Xie, Triantafyllos Afouras +3
The objective of this work is to localize sound sources that are visible in a video without using manual annotations. Our key technical contribution is to show that, by training th…
Self-supervised object detection from audio-visual correspondence
Triantafyllos Afouras, Yuki M. Asano, Francois Fagan +2
We tackle the problem of learning object detectors without supervision. Differently from weakly-supervised object detection, we do not assume image-level class labels. Instead, we…
Read and Attend: Temporal Localisation in Sign Language Videos
Gül Varol, Liliane Momeni, Samuel Albanie +2
The objective of this work is to annotate sign instances across a broad vocabulary in continuous sign language. We train a Transformer model to ingest a continuous signing stream a…
Watch, read and lookup: learning to spot signs from multiple supervisors
Liliane Momeni, Gül Varol, Samuel Albanie +2
The focus of this work is sign spotting - given a video of an isolated sign, our task is to identify whether and where it has been signed in a continuous, co-articulated sign langu…
Seeing wake words: Audio-visual Keyword Spotting
Liliane Momeni, Triantafyllos Afouras, Themos Stafylakis +2
The goal of this work is to automatically determine whether and when a word of interest is spoken by a talking face, with or without the audio. We propose a zero-shot method suitab…
Self-Supervised Learning of Audio-Visual Objects from Video
Triantafyllos Afouras, Andrew Owens, Joon Son Chung +1
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize…
BSL-1K: Scaling up co-articulated sign language recognition using mouthing cues
Samuel Albanie, Gül Varol, Liliane Momeni +4
Recent progress in fine-grained gesture and action classification, and machine translation, point to the possibility of automated sign language recognition becoming a reality. A ke…
Spot the conversation: speaker diarisation in the wild
Joon Son Chung, Jaesung Huh, Arsha Nagrani +2
The goal of this paper is speaker diarisation of videos collected 'in the wild'. We make three key contributions. First, we propose an automatic audio-visual diarisation method for…
ASR is all you need: cross-modal distillation for lip reading
Triantafyllos Afouras, Joon Son Chung, Andrew Zisserman
The goal of this work is to train strong models for visual speech recognition without requiring human annotated ground truth data. We achieve this by distilling from an Automatic S…