A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Abstract

With the development of deep learning and artificial intelligence, audio synthesis has a pivotal role in the area of machine learning and shows strong applicability in the industry. Meanwhile, significant efforts have been dedicated by researchers to handle multimodal tasks at present such as audio-visual multimodal processing. In this paper, we conduct a survey on audio synthesis and audio-visual multimodal processing, which helps understand current research and future trends. This review focuses on text to speech(TTS), music generation and some tasks that combine visual and acoustic information. The corresponding technical methods are comprehensively classified and introduced, and their future development trends are prospected. This survey can provide some guidance for researchers who are interested in the areas like audio synthesis and audio-visual multimodal processing.

References in corpus (27)

• Distilling the Knowledge in a Neural Network
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• VoxCeleb: a large-scale speaker identification dataset
• FastSpeech: Fast, Robust and Controllable Text to Speech
• FastSpeech 2: Fast and High-Quality End-to-End Text to Speech
• C-RNN-GAN: Continuous recurrent neural networks with adversarial training
• MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation
• Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search
• LipNet: End-to-End Sentence-level Lipreading
• A Survey on Neural Speech Synthesis
• MuseGAN: Multi-track Sequential Generative Adversarial Networks for Symbolic Music Generation and Accompaniment
• Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speech
• MelNet: A Generative Model for Audio in the Frequency Domain
• Jukebox: A Generative Model for Music
• Adversarial Audio Synthesis
• Generating Visually Aligned Sound from Videos
• Flowtron: an Autoregressive Flow-based Generative Network for Text-to-Speech Synthesis
• FaceFilter: Audio-visual speech separation using still images
• ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech
• Neural Speech Synthesis with Transformer Network
• Convolutional Generative Adversarial Networks with Binary Neurons for Polyphonic Music Generation
• Non-Homogeneous Haze Removal via Artificial Scene Prior and Bidimensional Graph Reasoning
• FloWaveNet : A Generative Flow for Raw Audio
• Arbitrary Talking Face Generation via Attentional Audio-Visual Coherence Learning
• Talking Face Generation by Conditional Recurrent Adversarial Network
• VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency
• LiRA: Learning Visual Speech Representations from Audio through Self-supervision