Sound2Sight: Generating Visual Dynamics from Sound and Context

Abstract

Learning associations across modalities is critical for robust multimodal reasoning, especially when a modality may be missing during inference. In this paper, we study this problem in the context of audio-conditioned visual synthesis -- a task that is important, for example, in occlusion reasoning. Specifically, our goal is to generate future video frames and their motion dynamics conditioned on audio and a few past frames. To tackle this problem, we present Sound2Sight, a deep variational framework, that is trained to learn a per frame stochastic prior conditioned on a joint embedding of audio and past frames. This embedding is learned via a multi-head attention-based audio-visual transformer encoder. The learned prior is then sampled to further condition a video forecasting module to generate future frames. The stochastic prior allows the model to sample multiple plausible futures that are consistent with the provided audio and the past context. Moreover, to improve the quality and coherence of the generated frames, we propose a multimodal discriminator that differentiates between a synthesized and a real audio-visual clip. We empirically evaluate our approach, vis-á-vis closely-related prior methods, on two new datasets viz. (i) Multimodal Stochastic Moving MNIST with a Surprise Obstacle, (ii) Youtube Paintings; as well as on the existing Audio-Set Drums dataset. Our extensive experiments demonstrate that Sound2Sight significantly outperforms the state of the art in the generated video quality, while also producing diverse video content.

Accepted at ECCV 2020

References in corpus (13)

• Adam: A Method for Stochastic Optimization
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Sequence to Sequence Learning with Neural Networks
• Unsupervised Learning of Video Representations using LSTMs
• Decomposing Motion and Content for Natural Video Sequence Prediction
• Stochastic Video Generation with a Learned Prior
• Video (language) modeling: a baseline for generative models of natural videos
• Video-to-Video Synthesis
• Learning Visual Predictive Models of Physics for Playing Billiards
• Stochastic Variational Video Prediction
• Sliced-Wasserstein Autoencoder: An Embarrassingly Simple Generative Model
• Discriminative Regularization for Generative Models
• End-to-End Speech-Driven Facial Animation with Temporal GANs