Generating Visually Aligned Sound from Videos

Abstract

We focus on the task of generating sound from natural videos, and the sound should be both temporally and content-wise aligned with visual signals. This task is extremely challenging because some sounds generated \emph{outside} a camera can not be inferred from video content. The model may be forced to learn an incorrect mapping between visual content and these irrelevant sounds. To address this challenge, we propose a framework named REGNET. In this framework, we first extract appearance and motion features from video frames to better distinguish the object that emits sound from complex background information. We then introduce an innovative audio forwarding regularizer that directly considers the real sound as input and outputs bottlenecked sound features. Using both visual and bottlenecked sound features for sound prediction during training provides stronger supervision for the sound prediction. The audio forwarding regularizer can control the irrelevant sound component and thus prevent the model from learning an incorrect mapping between video frames and sound emitted by the object that is out of the screen. During testing, the audio forwarding regularizer is removed to ensure that REGNET can produce purely aligned sound only from visual features. Extensive evaluations based on Amazon Mechanical Turk demonstrate that our method significantly improves both temporal and content-wise alignment. Remarkably, our generated sound can fool the human with a 68.12% success rate. Code and pre-trained models are publicly available at https://github.com/PeihaoChen/regnet

Published in IEEE Transactions on Image Processing, 2020. Code, pre-trained models and demo video: https://github.com/PeihaoChen/regnet

References in corpus (6)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Conditional Generative Adversarial Nets
• WaveNet: A Generative Model for Raw Audio
• SampleRNN: An Unconditional End-to-End Neural Audio Generation Model
• GANSynth: Adversarial Neural Audio Synthesis
• Gated-GAN: Adversarial Gated Networks for Multi-Collection Style Transfer

Cited by in corpus (9)

• Taming Visually Guided Sound Generation
• Multi-Instrumentalist Net: Unsupervised Generation of Music from Body Movements
• Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generation
• Soundify: Matching Sound Effects to Video
• Foley Music: Learning to Generate Music from Videos
• A Survey on Audio Synthesis and Audio-Visual Multimodal Processing
• Video Reenactment as Inductive Bias for Content-Motion Disentanglement
• Enhancing Sound Source Localization via False Negative Elimination
• SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations