3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video

Abstract

3D object reconstructions of transparent and concave structured objects, with inferred material properties, remains an open research problem for robot navigation in unstructured environments. In this paper, we propose a multimodal single- and multi-frame neural network for 3D reconstructions using audio-visual inputs. Our trained reconstruction LSTM autoencoder 3D-MOV accepts multiple inputs to account for a variety of surface types and views. Our neural network produces high-quality 3D reconstructions using voxel representation. Based on Intersection-over-Union (IoU), we evaluate against other baseline methods using synthetic audio-visual datasets ShapeNet and Sound20K with impact sounds and bounding box annotations. To the best of our knowledge, our single- and multi-frame model is the first audio-visual reconstruction neural network for 3D geometry and material representation.

References in corpus (16)

• Conditional Generative Adversarial Nets
• Deep Residual Learning for Image Recognition
• PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
• Conditional Image Synthesis With Auxiliary Classifier GANs
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• Learning a Probabilistic Latent Space of Object Shapes via 3D Generative-Adversarial Modeling
• Layer Normalization
• Learning What and Where to Draw
• Learning shape correspondence with anisotropic convolutional neural networks
• Multimodal Machine Learning: A Survey and Taxonomy
• Dive into Deep Learning
• Learning Temporal Regularity in Video Sequences
• ScanComplete: Large-Scale Scene Completion and Semantic Segmentation for 3D Scans
• Look, Listen and Learn
• MUREL: Multimodal Relational Reasoning for Visual Question Answering
• Customizing an Adversarial Example Generator with Class-Conditional GANs