297 citations · 601 across the 19 of their papers we have counts for
25 papers
Image Generators are Generalist Vision Learners
Valentin Gabeur, Shangbang Long, Songyou Peng +22
Recent works show that image and video generators exhibit zero-shot visual understanding behaviors, in a way reminiscent of how LLMs develop emergent capabilities of language under…
PaliGemma: A versatile 3B VLM for transfer
Lucas Beyer, Andreas Steiner, André Susano Pinto +32
PaliGemma is an open Vision-Language Model (VLM) that is based on the SigLIP-So400m vision encoder and the Gemma-2B language model. It is trained to be a versatile and broadly know…
Text Prompting for Multi-Concept Video Customization by Autoregressive Generation
Divya Kothandaraman, Kihyuk Sohn, Ruben Villegas +3
We present a method for multi-concept customization of pretrained text-to-video (T2V) models. Intuitively, the multi-concept customized video can be derived from the (non-linear) i…
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini Team, Petko Georgiev, Ving Ian Lei +1132
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over…
Point-VOS: Pointing Up Video Object Segmentation
Idil Esen Zulfikar, Sabarinath Mahadevan, Paul Voigtlaender +1
Current state-of-the-art Video Object Segmentation (VOS) methods rely on dense per-object mask annotations both during training and testing. This requires time-consuming and costly…
PaLI-3 Vision Language Models: Smaller, Faster, Stronger
Xi Chen, Xiao Wang, Lucas Beyer +16
This paper presents PaLI-3, a smaller, faster, and stronger vision language model (VLM) that compares favorably to similar models that are 10x larger. As part of arriving at this s…
StoryBench: A Multifaceted Benchmark for Continuous Story Visualization
Emanuele Bugliarello, Hernan Moraldo, Ruben Villegas +7
Generating video stories from text prompts is a complex task. In addition to having high visual quality, videos need to realistically adhere to a sequence of text prompts whilst be…
Connecting Vision and Language with Video Localized Narratives
Paul Voigtlaender, Soravit Changpinyo, Jordi Pont-Tuset +2
We propose Video Localized Narratives, a new form of multimodal video annotations connecting vision and language. In the original Localized Narratives, annotators speak and move th…
BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Video
Ali Athar, Jonathon Luiten, Paul Voigtlaender +4
Multiple existing benchmarks involve tracking and segmenting objects in video e.g., Video Object Segmentation (VOS) and Multi-Object Tracking and Segmentation (MOTS), but there is…
STEP: Segmenting and Tracking Every Pixel
Mark Weber, Jun Xie, Maxwell Collins +10
The task of assigning semantic classes and track identities to every pixel in a video is called video panoptic segmentation. Our work is the first that targets this task in a real-…
Reducing the Annotation Effort for Video Object Segmentation Datasets
Paul Voigtlaender, Lishu Luo, Chun Yuan +2
For further progress in video object segmentation (VOS), larger, more diverse, and more challenging datasets will be necessary. However, densely labeling every frame with pixel mas…
Siam R-CNN: Visual Tracking by Re-Detection
Paul Voigtlaender, Jonathon Luiten, Philip H. S. Torr +1
We present Siam R-CNN, a Siamese re-detection architecture which unleashes the full power of two-stage object detection approaches for visual object tracking. We combine this with…
Large-Scale Object Mining for Object Discovery from Unlabeled Video
Aljosa Osep, Paul Voigtlaender, Jonathon Luiten +2
This paper addresses the problem of object discovery from unlabeled driving videos captured in a realistic automotive setting. Identifying recurring object categories in such raw v…
BoLTVOS: Box-Level Tracking for Video Object Segmentation
Paul Voigtlaender, Jonathon Luiten, Bastian Leibe
We approach video object segmentation (VOS) by splitting the task into two sub-tasks: bounding box level tracking, followed by bounding box segmentation. Following this paradigm, w…
FEELVOS: Fast End-to-End Embedding Learning for Video Object Segmentation
Paul Voigtlaender, Yuning Chai, Florian Schroff +3
Many of the recent successful methods for video object segmentation (VOS) are overly complicated, heavily rely on fine-tuning on the first frame, and/or are slow, and are hence of…
MOTS: Multi-Object Tracking and Segmentation
Paul Voigtlaender, Michael Krause, Aljosa Osep +4
This paper extends the popular task of multi-object tracking to multi-object tracking and segmentation (MOTS). Towards this goal, we create dense pixel-level annotations for two ex…
4D Generic Video Object Proposals
Aljosa Osep, Paul Voigtlaender, Mark Weber +2
Many high-level video understanding methods require input in the form of object proposals. Currently, such proposals are predominantly generated with the help of networks that were…
Towards Large-Scale Video Video Object Mining
Aljosa Osep, Paul Voigtlaender, Jonathon Luiten +2
We propose to leverage a generic object tracker in order to perform object mining in large-scale unlabeled videos, captured in a realistic automotive setting. We present a dataset…
PReMVOS: Proposal-generation, Refinement and Merging for Video Object Segmentation
Jonathon Luiten, Paul Voigtlaender, Bastian Leibe
We address semi-supervised video object segmentation, the task of automatically generating accurate and consistent pixel masks for objects in a video sequence, given the first-fram…
Iteratively Trained Interactive Segmentation
Sabarinath Mahadevan, Paul Voigtlaender, Bastian Leibe
Deep learning requires large amounts of training data to be effective. For the task of object segmentation, manually labeling data is very expensive, and hence interactive methods…
Large-Scale Object Discovery and Detector Adaptation from Unlabeled Video
Aljoša Ošep, Paul Voigtlaender, Jonathon Luiten +2
We explore object discovery and detector adaptation based on unlabeled video sequences captured from a mobile platform. We propose a fully automatic approach for object mining from…
Track, then Decide: Category-Agnostic Vision-based Multi-Object Tracking
Aljoša Ošep, Wolfgang Mehner, Paul Voigtlaender +1
The most common paradigm for vision-based multi-object tracking is tracking-by-detection, due to the availability of reliable detectors for several important object categories such…
Online Adaptation of Convolutional Neural Networks for Video Object Segmentation
Paul Voigtlaender, Bastian Leibe
We tackle the task of semi-supervised video object segmentation, i.e. segmenting the pixels belonging to an object in the video using the ground truth pixel mask for the first fram…
RETURNN: The RWTH Extensible Training framework for Universal Recurrent Neural Networks
Patrick Doetsch, Albert Zeyer, Paul Voigtlaender +3
In this work we release our extensible and easily configurable neural network training software. It provides a rich set of functional layers with a particular focus on efficient tr…