53 citations · 162 across the 33 of their papers we have counts for
39 papers
Memory-Guided B-Roll Generation from User Video Collections
Cusuh Ham, Fabian Caba Heilbron, Josef Sivic +1
We introduce an approach for collection-grounded B-roll sequence generation. Given a user's video collection, a directive given in natural language, and a target duration, the goal…
Controlling Embedding Spaces with Text-Conditioned Transformations
Joseph Fioresi, Fabian Caba Heilbron, Pankaj Nathani +2
Multimodal embedding spaces in models like CLIP enable powerful capabilities such as semantic similarity retrieval and cross-modal zero-shot classification. These embeddings compre…
Autoregressive Modeling of Film with Applications in Video Montage
Marcelo Sandoval-Castañeda, Fabian Caba Heilbron, Shiry Ginosar +4
This work introduces FilmGPT, an autoregressive transformer designed to address the challenge of video montage--turning a collection of raw, "unwatchable" footage into coherent cin…
Gen2Balance: Generative Balancing for Long-Tailed Video Action Recognition
Prajwal Gatti, Simon Jenni, Fabian Caba Heilbron +1
We address the problem of training on long-tailed data for video action recognition. We propose to augment the training set using a text-to-video generative model, conditioned on d…
ResidualViT for Efficient Temporally Dense Video Encoding
Mattia Soldan, Fabian Caba Heilbron, Bernard Ghanem +2
Several video understanding tasks, such as natural language temporal video grounding, temporal activity localization, and audio description generation, require "temporally dense" r…
Discovering Divergent Representations between Text-to-Image Models
Lisa Dunlap, Joseph E. Gonzalez, Trevor Darrell +3
In this paper, we investigate when and how visual representations learned by two different generative models diverge. Given two text-to-image models, our goal is to discover visual…
Improving Personalized Search with Regularized Low-Rank Parameter Updates
Fiona Ryan, Josef Sivic, Fabian Caba Heilbron +3
Personalized vision-language retrieval seeks to recognize new concepts (e.g. "my dog Fido") from only a few examples. This task is challenging because it requires not only learning…
CineVerse: Consistent Keyframe Synthesis for Cinematic Scene Composition
Quynh Phung, Long Mai, Fabian David Caba Heilbron +3
We present CineVerse, a novel framework for the task of cinematic scene composition. Similar to traditional multi-shot generation, our task emphasizes the need for consistency and…
Generative Timelines for Instructed Visual Assembly
Alejandro Pardo, Jui-Hsien Wang, Bernard Ghanem +3
The objective of this work is to manipulate visual timelines (e.g. a video) through natural language instructions, making complex timeline editing tasks accessible to non-expert or…
Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets
Ishan Rajendrakumar Dave, Fabian Caba Heilbron, Mubarak Shah +1
Temporal video alignment aims to synchronize the key events like object interactions or action phase transitions in two videos. Such methods could benefit various video editing, pr…
Adapting Dual-encoder Vision-language Models for Paraphrased Retrieval
Jiacheng Cheng, Hijung Valentina Shin, Nuno Vasconcelos +2
In the recent years, the dual-encoder vision-language models (\eg CLIP) have achieved remarkable text-to-image retrieval performance. However, we discover that these models usually…
Concept Weaver: Enabling Multi-Concept Fusion in Text-to-Image Models
Gihyun Kwon, Simon Jenni, Dingzeyu Li +3
While there has been significant progress in customizing text-to-image generation models, generating images that combine multiple personalized concepts remains challenging. In this…
Towards Automated Movie Trailer Generation
Dawit Mureja Argaw, Mattia Soldan, Alejandro Pardo +4
Movie trailers are an essential tool for promoting films and attracting audiences. However, the process of creating trailers can be time-consuming and expensive. To streamline this…
Scaling Up Video Summarization Pretraining with Large Language Models
Dawit Mureja Argaw, Seunghyun Yoon, Fabian Caba Heilbron +5
Long-form video content constitutes a significant portion of internet traffic, making automated video summarization an essential research problem. However, existing video summariza…
Long-range Multimodal Pretraining for Movie Understanding
Dawit Mureja Argaw, Joon-Young Lee, Markus Woodson +2
Learning computer vision models from (and for) movies has a long-standing history. While great progress has been attained, there is still a need for a pretrained multimodal model t…
Meta-Personalizing Vision-Language Models to Find Named Instances in Video
Chun-Hsiao Yeh, Bryan Russell, Josef Sivic +2
Large-scale vision-language models (VLM) have shown impressive results for language-guided search applications. While these models allow category-level queries, they currently stru…
PIVOT: Prompting for Video Continual Learning
Andrés Villa, Juan León Alcázar, Motasem Alfarra +5
Modern machine learning pipelines are limited due to data availability, storage quotas, privacy regulations, and expensive annotation processes. These constraints make it difficult…
VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Space
David Chuan-En Lin, Fabian Caba Heilbron, Joon-Young Lee +2
Video editing is a creative and complex endeavor and we believe that there is potential for reimagining a new video editing interface to better support the creative and exploratory…
The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing
Dawit Mureja Argaw, Fabian Caba Heilbron, Joon-Young Lee +2
Machine learning is transforming the video editing industry. Recent advances in computer vision have leveled-up video editing tasks such as intelligent reframing, rotoscoping, colo…
Video-ReTime: Learning Temporally Varying Speediness for Time Remapping
Simon Jenni, Markus Woodson, Fabian Caba Heilbron
We propose a method for generating a temporally remapped video that matches the desired target duration while maximally preserving natural video dynamics. Our approach trains a neu…
FitCLIP: Refining Large-Scale Pretrained Image-Text Models for Zero-Shot Video Understanding Tasks
Santiago Castro, Fabian Caba Heilbron
Large-scale pretrained image-text models have shown incredible zero-shot performance in a handful of tasks, including video ones such as action recognition and text-to-video retrie…
OWL (Observe, Watch, Listen): Audiovisual Temporal Context for Localizing Actions in Egocentric Videos
Merey Ramazanova, Victor Escorcia, Fabian Caba Heilbron +2
Egocentric videos capture sequences of human activities from a first-person perspective and can provide rich multimodal signals. However, most current localization methods use thir…
vCLIMB: A Novel Video Class Incremental Learning Benchmark
Andrés Villa, Kumail Alhamoud, Juan León Alcázar +3
Continual learning (CL) is under-explored in the video domain. The few existing works contain splits with imbalanced class distributions over the tasks, or study the problem in uns…
MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptions
Mattia Soldan, Alejandro Pardo, Juan León Alcázar +4
The recent and increasing interest in video-language research has driven the development of large-scale datasets that enable data-intensive machine learning techniques. In comparis…