81 citations · 263 across the 27 of their papers we have counts for
18 papers · 1 filter
Cosmos 3: Omnimodal World Models for Physical AI
NVIDIA, :, Aditi +293
We introduce Cosmos 3, a family of omnimodal world models designed to jointly process and generate language, image, video, audio, and action sequences within a unified mixture-of-t…
Benchmarking Single-Factor Physical Video-to-Audio Generation
Tingle Li, Siddharth Gururani, Kevin J. Shih +6
Generative video-to-audio (V2A) models produce highly plausible soundtracks, but it remains unclear whether they capture the underlying physical processes. Existing evaluations emp…
Enhancing Virtual Try-On with Synthetic Pairs and Error-Aware Noise Scheduling
Nannan Li, Kevin J. Shih, Bryan A. Plummer
Given an isolated garment image in a canonical product view and a separate image of a person, the virtual try-on task aims to generate a new image of the person wearing the target…
Collecting The Puzzle Pieces: Disentangled Self-Driven Human Pose Transfer by Permuting Textures
Nannan Li, Kevin J. Shih, Bryan A. Plummer
Human pose transfer synthesizes new view(s) of a person for a given pose. Recent work achieves this via self-reconstruction, which disentangles a person's pose and texture informat…
Unsupervised Disentanglement of Pose, Appearance and Background from Images and Videos
Aysegul Dundar, Kevin J. Shih, Animesh Garg +3
Unsupervised landmark learning is the task of learning semantic keypoint-like representations without the use of expensive input keypoint-level annotations. A popular approach is t…
Video Interpolation and Prediction with Unsupervised Landmarks
Kevin J. Shih, Aysegul Dundar, Animesh Garg +3
Prediction and interpolation for long-range video data involves the complex task of modeling motion trajectories for each visible object, occlusions and dis-occlusions, as well as…
Unsupervised Video Interpolation Using Cycle Consistency
Fitsum A. Reda, Deqing Sun, Aysegul Dundar +6
Learning to synthesize high frame rate videos via interpolation requires large quantities of high frame rate training videos, which, however, are scarce, especially at high resolut…
Graphical Contrastive Losses for Scene Graph Parsing
Ji Zhang, Kevin J. Shih, Ahmed Elgammal +2
Most scene graph parsers use a two-stage pipeline to detect visual relationships: the first stage detects entities, and the second predicts the predicate for each entity pair using…
Improving Semantic Segmentation via Video Propagation and Label Relaxation
Yi Zhu, Karan Sapra, Fitsum A. Reda +4
Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training…
Partial Convolution based Padding
Guilin Liu, Kevin J. Shih, Ting-Chun Wang +5
In this paper, we present a simple yet effective padding scheme that can be used as a drop-in module for existing convolutional neural networks. We call it partial convolution base…
An Interpretable Model for Scene Graph Generation
Ji Zhang, Kevin Shih, Andrew Tao +2
We propose an efficient and interpretable scene graph generator. We consider three types of features: visual, spatial and semantic, and we use a late fusion strategy such that each…
Introduction to the 1st Place Winning Model of OpenImages Relationship Detection Challenge
Ji Zhang, Kevin Shih, Andrew Tao +2
This article describes the model we built that achieved 1st place in the OpenImage Visual Relationship Detection Challenge on Kaggle. Three key factors contribute the most to our s…
Revisiting Image-Language Networks for Open-ended Phrase Detection
Bryan A. Plummer, Kevin J. Shih, Yichen Li +4
Most existing work that grounds natural language phrases in images starts with the assumption that the phrase in question is relevant to the image. In this paper we address a more…
SDCNet: Video Prediction Using Spatially-Displaced Convolution
Fitsum A. Reda, Guilin Liu, Kevin J. Shih +5
We present an approach for high-resolution video frame prediction by conditioning on both past frames and past optical flows. Previous approaches rely on resampling past frames, gu…
Image Inpainting for Irregular Holes Using Partial Convolutions
Guilin Liu, Fitsum A. Reda, Kevin J. Shih +3
Existing deep learning based image inpainting methods use a standard convolutional network over the corrupted image, using convolutional filter responses conditioned on both valid…
Aligned Image-Word Representations Improve Inductive Transfer Across Vision-Language Tasks
Tanmay Gupta, Kevin Shih, Saurabh Singh +1
An important goal of computer vision is to build systems that learn visual representations over time that can be applied to many tasks. In this paper, we investigate a vision-langu…
Where To Look: Focus Regions for Visual Question Answering
Kevin J. Shih, Saurabh Singh, Derek Hoiem
We present a method that learns to answer visual questions by selecting image regions relevant to the text-based query. Our method exhibits significant improvements in answering qu…
Part Localization using Multi-Proposal Consensus for Fine-Grained Categorization
Kevin J. Shih, Arun Mallya, Saurabh Singh +1
We present a simple deep learning framework to simultaneously predict keypoint locations and their respective visibilities and use those to achieve state-of-the-art performance for…