374 citations · 944 across the 24 of their papers we have counts for
30 papers
Conditional Generation of Audio from Video via Foley Analogies
Yuexi Du, Ziyang Chen, Justin Salamon +2
The sound effects that designers add to videos are designed to convey a particular artistic effect and, thus, may be quite different from a scene's true sound. Inspired by the chal…
Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment
Kim Sung-Bin, Arda Senocak, Hyunwoo Ha +2
How does audio describe the world around us? In this paper, we propose a method for generating an image of a scene from sound. Our method addresses the challenges of dealing with t…
Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Models
Lukas Höllein, Ang Cao, Andrew Owens +2
We present Text2Room, a method for generating room-scale textured 3D meshes from a given text prompt as input. To this end, we leverage pre-trained 2D text-to-image models to synth…
Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation
Ziyang Chen, Shengyi Qian, Andrew Owens
The images and sounds that we perceive undergo subtle but geometrically consistent changes as we rotate our heads. In this paper, we use these cues to solve a problem we call Sound…
EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata
Chenhao Zheng, Ayush Shrivastava, Andrew Owens
We learn a visual representation that captures information about the camera that recorded a given photo. To do this, we train a multimodal embedding between image patches and the E…
Self-Supervised Video Forensics by Audio-Visual Anomaly Detection
Chao Feng, Ziyang Chen, Andrew Owens
Manipulated videos often contain subtle inconsistencies between their visual and audio signals. We propose a video forensics method, based on anomaly detection, that can identify t…
Touch and Go: Learning from Human-Collected Vision and Touch
Fengyu Yang, Chenyang Ma, Jiacheng Zhang +3
The ability to associate touch with sight is essential for tasks that require physically interacting with objects in the world. We propose a dataset with paired visual and tactile…
Mix and Localize: Localizing Sound Sources in Mixtures
Xixi Hu, Ziyang Chen, Andrew Owens
We present a method for simultaneously localizing multiple sound sources within a visual scene. This task requires a model to both group a sound mixture into individual sources, an…
Learning Visual Styles from Audio-Visual Associations
Tingle Li, Yichen Liu, Andrew Owens +1
From the patter of rain to the crunch of snow, the sounds we hear often convey the visual textures that appear within a scene. In this paper, we present a method for learning visua…
Sound Localization by Self-Supervised Time Delay Estimation
Ziyang Chen, David F. Fouhey, Andrew Owens
Sounds reach one microphone in a stereo pair sooner than the other, resulting in an interaural time delay that conveys their directions. Estimating a sound's time delay requires fi…
Learning Pixel Trajectories with Multiscale Contrastive Random Walks
Zhangxing Bian, Allan Jabri, Alexei A. Efros +1
A range of video modeling tasks, from optical flow to multiple object tracking, share the same fundamental challenge: establishing space-time correspondence. Yet, approaches that d…
GANmouflage: 3D Object Nondetection with Texture Fields
Rui Guo, Jasmine Collins, Oscar de Lima +1
We propose a method that learns to camouflage 3D objects within scenes. Given an object's shape and a distribution of viewpoints from which it will be seen, we estimate a texture t…
Structure from Silence: Learning Scene Structure from Ambient Sound
Ziyang Chen, Xixi Hu, Andrew Owens
From whirling ceiling fans to ticking clocks, the sounds that we hear subtly vary as we move through a scene. We ask whether these ambient sounds convey information about 3D scene…
Strumming to the Beat: Audio-Conditioned Contrastive Video Textures
Medhini Narasimhan, Shiry Ginosar, Andrew Owens +2
We introduce a non-parametric approach for infinite video texture synthesis using a representation learned via contrastive learning. We take inspiration from Video Textures, which…
Comparing Correspondences: Video Prediction with Correspondence-wise Losses
Daniel Geng, Max Hamilton, Andrew Owens
Image prediction methods often struggle on tasks that require changing the positions of objects, such as video prediction, producing blurry images that average over the many positi…
Planar Surface Reconstruction from Sparse Views
Linyi Jin, Shengyi Qian, Andrew Owens +1
The paper studies planar surface reconstruction of indoor scenes from two views with unknown camera poses. While prior approaches have successfully created object-centric reconstru…
Self-Supervised Learning of Audio-Visual Objects from Video
Triantafyllos Afouras, Andrew Owens, Joon Son Chung +1
Our objective is to transform a video into a set of discrete audio-visual objects using self-supervised learning. To this end, we introduce a model that uses attention to localize…
Space-Time Correspondence as a Contrastive Random Walk
Allan Jabri, Andrew Owens, Alexei A. Efros
This paper proposes a simple self-supervised approach for learning a representation for visual correspondence from raw video. We cast correspondence as prediction of links in a spa…
CNN-generated images are surprisingly easy to spot... for now
Sheng-Yu Wang, Oliver Wang, Richard Zhang +2
In this work we ask whether it is possible to create a "universal" detector for telling apart real images from these generated by a CNN, regardless of architecture or dataset used.…
Learning Individual Styles of Conversational Gesture
Shiry Ginosar, Amir Bar, Gefen Kohavi +3
Human speech is often accompanied by hand and arm gestures. Given audio speech input, we generate plausible gestures to go along with the sound. Specifically, we perform cross-moda…
Detecting Photoshopped Faces by Scripting Photoshop
Sheng-Yu Wang, Oliver Wang, Andrew Owens +2
Most malicious photo manipulations are created using standard image editing tools, such as Adobe Photoshop. We present a method for detecting one very popular Photoshop manipulatio…
MoSculp: Interactive Visualization of Shape and Time
Xiuming Zhang, Tali Dekel, Tianfan Xue +5
We present a system that allows users to visualize complex human motion via 3D motion sculptures---a representation that conveys the 3D structure swept by a human body as it moves…
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch
Roberto Calandra, Andrew Owens, Dinesh Jayaraman +5
For humans, the process of grasping an object relies heavily on rich tactile feedback. Most recent robotic grasping work, however, has been based only on visual input, and thus can…
Fighting Fake News: Image Splice Detection via Learned Self-Consistency
Minyoung Huh, Andrew Liu, Andrew Owens +1
Advances in photo editing and manipulation tools have made it significantly easier to create fake imagery. Learning to detect such manipulations, however, remains a challenging pro…