127 citations · 162 across the 10 of their papers we have counts for
13 papers
Finding Fallen Objects Via Asynchronous Audio-Visual Integration
Chuang Gan, Yi Gu, Siyuan Zhou +7
The way an object looks and sounds provide complementary reflections of its physical properties. In many settings cues from vision and audition arrive asynchronously but must be in…
Object-based synthesis of scraping and rolling sounds based on non-linear physical constraints
Vinayak Agarwal, Maddie Cusimano, James Traer +1
Sustained contact interactions like scraping and rolling produce a wide variety of sounds. Previous studies have explored ways to synthesize these sounds efficiently and intuitivel…
Neural Population Geometry Reveals the Role of Stochasticity in Robust Perception
Joel Dapello, Jenelle Feather, Hang Le +5
Adversarial examples are often cited by neuroscientists and machine learning researchers as an example of how computational models diverge from biological sensory systems. Recent w…
The ThreeDWorld Transport Challenge: A Visually Guided Task-and-Motion Planning Benchmark for Physically Realistic Embodied AI
Chuang Gan, Siyuan Zhou, Jeremy Schwartz +8
We introduce a visually-guided and physics-driven task-and-motion planning benchmark, which we call the ThreeDWorld Transport Challenge. In this challenge, an embodied agent equipp…
Speech Denoising with Auditory Models
Mark R. Saddler, Andrew Francl, Jenelle Feather +3
Contemporary speech enhancement predominantly relies on audio transforms that are trained to reconstruct a clean speech waveform. The development of high-performing neural network…
ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation
Chuang Gan, Jeremy Schwartz, Seth Alter +21
We introduce ThreeDWorld (TDW), a platform for interactive multi-modal physical simulation. TDW enables simulation of high-fidelity sensory data and physical interactions between m…
Untangling in Invariant Speech Recognition
Cory Stephenson, Jenelle Feather, Suchismita Padhy +4
Encouraged by the success of deep neural networks on a variety of visual tasks, much theoretical and experimental work has been aimed at understanding and interpreting how vision n…
Self-Supervised Audio-Visual Co-Segmentation
Andrew Rouditchenko, Hang Zhao, Chuang Gan +2
Segmenting objects in images and separating sound sources in audio are challenging tasks, in part because traditional approaches require large amounts of labeled data. In this pape…
The Sound of Pixels
Hang Zhao, Chuang Gan, Andrew Rouditchenko +3
We introduce PixelPlayer, a system that, by leveraging large amounts of unlabeled videos, learns to locate image regions which produce sounds and separate the input sounds into a s…
Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott +2
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds ca…
Learning Mid-Level Auditory Codes from Natural Sound Statistics
Wiktor Młynarski, Josh H. McDermott
Interaction with the world requires an organism to transform sensory signals into representations in which behaviorally meaningful properties of the environment are made explicit.…
Ambient Sound Provides Supervision for Visual Learning
Andrew Owens, Jiajun Wu, Josh H. McDermott +2
The sound of crashing waves, the roar of fast-moving cars -- sound conveys important information about the objects in our surroundings. In this work, we show that ambient sounds ca…
Visually Indicated Sounds
Andrew Owens, Phillip Isola, Josh McDermott +3
Objects make distinctive sounds when they are hit or scratched. These sounds reveal aspects of an object's material properties, as well as the actions that produced them. In this p…