28 citations · 180 across the 24 of their papers we have counts for
24 papers
RealImpact: A Dataset of Impact Sound Fields for Real Objects
Samuel Clarke, Ruohan Gao, Mason Wang +5
Objects make unique sounds under different perturbations, environment conditions, and poses relative to the listener. While prior works have modeled impact sounds and sound propaga…
The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects
Ruohan Gao, Yiming Dou, Hao Li +5
We introduce the ObjectFolder Benchmark, a benchmark suite of 10 tasks for multisensory object-centric learning, centered around object recognition, reconstruction, and manipulatio…
Sonicverse: A Multisensory Simulation Platform for Embodied Household Agents that See and Hear
Ruohan Gao, Hao Li, Gokul Dharan +6
Developing embodied agents in simulation has been a key research topic in recent years. Exciting new tasks, algorithms, and benchmarks have been developed in various simulators. Ho…
An Extensible Multimodal Multi-task Object Dataset with Materials
Trevor Standley, Ruohan Gao, Dawn Chen +2
We present EMMa, an Extensible, Multimodal dataset of Amazon product listings that contains rich Material annotations. It contains more than 2.8 million objects, each with image(s)…
Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition
Hong-Xing Yu, Michelle Guo, Alireza Fathi +5
Photorealistic object appearance modeling from 2D images is a constant topic in vision and graphics. While neural implicit methods (such as Neural Radiance Fields) have shown high-…
See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation
Hao Li, Yizhi Zhang, Junzhe Zhu +7
Humans use all of their senses to accomplish different tasks in everyday activities. In contrast, existing work on robotic manipulation mostly relies on one, or occasionally two mo…
Differentiable Physics Simulation of Dynamics-Augmented Neural Objects
Simon Le Cleac'h, Hong-Xing Yu, Michelle Guo +5
We present a differentiable pipeline for simulating the motion of objects that represent their geometry as a continuous density field parameterized as a deep network. This includes…
Expected geoneutrino signal at JUNO using local integrated 3-D refined crustal model
Ran Han, ZhiWei Li, Ruohan Gao +11
Geoneutrinos serve as a potent tool for comprehending the radiogenic power and composition of Earth. Although geoneutrinos have been observed in prior experiments, the forthcoming…
ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer
Ruohan Gao, Zilin Si, Yen-Yu Chang +5
Objects play a crucial role in our everyday activities. Though multisensory object-centric learning has shown great potential lately, the modeling of objects in prior work is rathe…
Visual Acoustic Matching
Changan Chen, Ruohan Gao, Paul Calamia +1
We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment. Given an image of the target environmen…
Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Video
Rishabh Garg, Ruohan Gao, Kristen Grauman
Binaural audio provides human listeners with an immersive spatial sound experience, but most existing videos lack binaural audio recordings. We propose an audio spatialization meth…
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
Ruohan Gao, Yen-Yu Chang, Shivani Mall +2
Multisensory object-centric perception, reasoning, and interaction have been a key research topic in recent years. However, the progress in these directions is limited by the small…
VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
Ruohan Gao, Kristen Grauman
We introduce a new approach for audio-visual speech separation. Given a video, the goal is to extract the speech associated with a face in spite of simultaneous background sounds a…
Learning to Set Waypoints for Audio-Visual Navigation
Changan Chen, Sagnik Majumder, Ziad Al-Halah +3
In audio-visual navigation, an agent intelligently travels through a complex, unmapped 3D environment using both sights and sounds to find a sound source (e.g., a phone ringing in…
VisualEchoes: Spatial Image Representation Learning through Echolocation
Ruohan Gao, Changan Chen, Ziad Al-Halah +2
Several animal species (e.g., bats, dolphins, and whales) and even visually impaired humans have the remarkable ability to perform echolocation: a biological sonar used to perceive…
Listen to Look: Action Recognition by Previewing Audio
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman +1
In the face of the video data deluge, today's expensive clip-level classifiers are increasingly impractical. We propose a framework for efficient action recognition in untrimmed vi…
Co-Separating Sounds of Visual Objects
Ruohan Gao, Kristen Grauman
Learning how objects sound from video is challenging, since they often heavily overlap in a single audio channel. Current methods for visually-guided audio source separation sidest…
JULOC: A Local 3-D Refined Crust Model for the Geoneutrino Measurement at JUNO
Ruohan Gao, Zhiwei Li, Ran Han +7
Geothermal energy is the key to drive the plate tectonics and interior thermodynamics of the Earth. The surface heat flux, as measured in boreholes, provide limited insights into t…
2.5D Visual Sound
Ruohan Gao, Kristen Grauman
Binaural audio provides a listener with 3D sound sensation, allowing a rich perceptual experience of the scene. However, binaural recordings are scarcely available and require nont…
Learning to Separate Object Sounds by Watching Unlabeled Video
Ruohan Gao, Rogerio Feris, Kristen Grauman
Perceiving a scene most fully requires all the senses. Yet modeling how objects look and sound is challenging: most natural scenes and events contain multiple objects, and the audi…
Im2Flow: Motion Hallucination from Static Images for Action Recognition
Ruohan Gao, Bo Xiong, Kristen Grauman
Existing methods to recognize actions in static images take the images at their face value, learning the appearances---objects, scenes, and body poses---that distinguish each actio…
ShapeCodes: Self-Supervised Feature Learning by Lifting Views to Viewgrids
Dinesh Jayaraman, Ruohan Gao, Kristen Grauman
We introduce an unsupervised feature learning approach that embeds 3D shape information into a single-view image representation. The main idea is a self-supervised training objecti…
On-Demand Learning for Deep Image Restoration
Ruohan Gao, Kristen Grauman
While machine learning approaches to image restoration offer great promise, current methods risk training models fixated on performing well only for image corruption of a particula…
Object-Centric Representation Learning from Unlabeled Videos
Ruohan Gao, Dinesh Jayaraman, Kristen Grauman
Supervised (pre-)training currently yields state-of-the-art performance for representation learning for visual recognition, yet it comes at the cost of (1) intensive manual annotat…