Controllable Data Generation by Deep Learning: A Review
arXiv:2207.09542 · doi:10.1145/3648609
Abstract
Designing and generating new data under targeted properties has been attracting various critical applications such as molecule design, image editing and speech synthesis. Traditional hand-crafted approaches heavily rely on expertise experience and intensive human efforts, yet still suffer from the insufficiency of scientific knowledge and low throughput to support effective and efficient data generation. Recently, the advancement of deep learning has created the opportunity for expressive methods to learn the underlying representation and properties of data. Such capability provides new ways of determining the mutual relationship between the structural patterns and functional properties of the data and leveraging such relationships to generate structural data, given the desired properties. This article is a systematic review that explains this promising research area, commonly known as controllable deep data generation. First, the article raises the potential challenges and provides preliminaries. Then the article formally defines controllable deep data generation, proposes a taxonomy on various techniques and summarizes the evaluation metrics in this specific domain. After that, the article introduces exciting applications of controllable deep data generation, experimentally analyzes and compares existing works. Finally, this article highlights the promising future directions of controllable deep data generation and identifies five potential challenges.
arXiv commentThis survey has been accepted by ACM Computing Surveys
References in corpus (59)
- Deep Learning in Neural Networks: An Overview
- Conditional Generative Adversarial Nets
- A Brief Survey of Deep Reinforcement Learning
- Automatic chemical design using a data-driven continuous representation of molecules
- Diffusion Models Beat GANs on Image Synthesis
- Score-Based Generative Modeling through Stochastic Differential Equations
- Generative Modeling by Estimating Gradients of the Data Distribution
- GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Classifier-Free Diffusion Guidance
- Deep learning for molecular design - a review of the state of the art
- FastSpeech: Fast, Robust and Controllable Text to Speech
- Data Synthesis based on Generative Adversarial Networks
- GANSpace: Discovering Interpretable GAN Controls
- MolGAN: An implicit generative model for small molecular graphs
- Self-Supervised Representation Learning: Introduction, Advances and Challenges
- Learning Deep Generative Models of Graphs
- StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows
- Guided Image Generation with Conditional Invertible Neural Networks
- Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation
- Graph Representation Learning: A Survey
- MoFlow: An Invertible Flow Model for Generating Molecular Graphs
- CIAGAN: Conditional Identity Anonymization Generative Adversarial Networks
- A Survey on Neural Speech Synthesis
- Computational Protein Design with Deep Learning Neural Networks
- Human Motion Diffusion Model
- Discovering Hidden Factors of Variation in Deep Networks
- A survey on text generation using generative adversarial networks
- MelNet: A Generative Model for Audio in the Frequency Domain
- GRATIS: GeneRAting TIme Series with diverse and controllable characteristics
- GraphNVP: An Invertible Flow Model for Generating Molecular Graphs
- Controllable Neural Story Plot Generation via Reward Shaping
- Symmetry-adapted generation of 3d point sets for the targeted discovery of molecules
- Constrained Generation of Semantically Valid Graphs via Regularizing Variational Autoencoders
- Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization
- Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis
- Unsupervised Text Style Transfer using Language Models as Discriminators
- Generating Multi-Categorical Samples with Generative Adversarial Networks
- Multi-reference Tacotron by Intercross Training for Style Disentangling,Transfer and Control in Speech Synthesis
- An Application of Generative Adversarial Networks for Super Resolution Medical Imaging
- Deep Generative Models with Learnable Knowledge Constraints
- A Survey of Inductive Biases for Factorial Representation-Learning
- Latent Molecular Optimization for Targeted Therapeutic Design
- GAUCHE: A Library for Gaussian Processes in Chemistry
- Guiding Deep Molecular Optimization with Genetic Exploration
- Interpretable Deep Graph Generation with Node-Edge Co-Disentanglement
- MolGenSurvey: A Systematic Survey in Machine Learning Models for Molecule Design
- Review of end-to-end speech synthesis technology based on deep learning
- TG-GAN: Continuous-time Temporal Graph Generation with Deep Generative Models
- GraphEBM: Molecular Graph Generation with Energy-Based Models
- A Survey on Deep Graph Generation: Methods and Applications
- Model architectures to extrapolate emotional expressions in DNN-based text-to-speech
- C5T5: Controllable Generation of Organic Molecules with Transformers
- Deep Generative Model for Periodic Graphs
- AAG-Stega: Automatic Audio Generation-based Steganography
- MEGATRON-CNTRL: Controllable Story Generation with External Knowledge Using Large-Scale Language Models
- Deep Generation of Heterogeneous Networks
- Enhancing Diffusion-Based Image Synthesis with Robust Classifier Guidance
- Speech Synthesis and Control Using Differentiable DSP