A survey of synthetic data augmentation methods in computer vision

Abstract

The standard approach to tackling computer vision problems is to train deep convolutional neural network (CNN) models using large-scale image datasets which are representative of the target task. However, in many scenarios, it is often challenging to obtain sufficient image data for the target task. Data augmentation is a way to mitigate this challenge. A common practice is to explicitly transform existing images in desired ways so as to create the required volume and variability of training data necessary to achieve good generalization performance. In situations where data for the target domain is not accessible, a viable workaround is to synthesize training data from scratch--i.e., synthetic data augmentation. This paper presents an extensive review of synthetic data augmentation techniques. It covers data synthesis approaches based on realistic 3D graphics modeling, neural style transfer (NST), differential neural rendering, and generative artificial intelligence (AI) techniques such as generative adversarial networks (GANs) and variational autoencoders (VAEs). For each of these classes of methods, we focus on the important data generation and augmentation techniques, general scope of application and specific use-cases, as well as existing limitations and possible workarounds. Additionally, we provide a summary of common synthetic datasets for training computer vision models, highlighting the main features, application domains and supported tasks. Finally, we discuss the effectiveness of synthetic data augmentation methods. Since this is the first paper to explore synthetic data augmentation methods in great detail, we are hoping to equip readers with the necessary background information and in-depth knowledge of existing methods and their attendant issues.

References in corpus (60)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation
• GAN-based Synthetic Medical Image Augmentation for increased CNN Performance in Liver Lesion Classification
• Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks
• InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets
• Learning from Synthetic Humans
• Training Generative Adversarial Networks with Limited Data
• Learning to Discover Cross-Domain Relations with Generative Adversarial Networks
• Autoencoding beyond pixels using a learned similarity metric
• ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness
• Smart Augmentation - Learning an Optimal Data Augmentation Strategy
• GAN Augmentation: Augmenting Training Data using Generative Adversarial Networks
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Implicit Neural Representations with Periodic Activation Functions
• Looking Beyond Appearances: Synthetic Training Data for Deep CNNs in Re-identification
• Stable and Controllable Neural Texture Synthesis and Style Transfer Using Histogram Losses
• MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
• Artistic style transfer for videos
• Learning to Compose Domain-Specific Transformations for Data Augmentation
• FusionNet: 3D Object Classification Using Multiple Data Representations
• Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks
• Points2Surf: Learning Implicit Surfaces from Point Cloud Patches
• Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing
• A Survey on Face Data Augmentation
• Artistic style transfer for videos and spherical images
• An Arbitrary Scale Super-Resolution Approach for 3D MR Images via Implicit Neural Representation
• Laplacian-Steered Neural Style Transfer
• Occlusion Handling in Generic Object Detection: A Review
• Unity Perception: Generate Synthetic Data for Computer Vision
• Uncertainty Modeling for Out-of-Distribution Generalization
• Enhance to Read Better: A Multi-Task Adversarial Network for Handwritten Document Image Enhancement
• Transport-Based Neural Style Transfer for Smoke Simulations
• Facial-Sketch Synthesis: A New Challenge
• Consistency Regularization for Generative Adversarial Networks
• Object Detection Using Deep CNNs Trained on Synthetic Images
• DFA-NeRF: Personalized Talking Head Generation via Disentangled Face Attributes Neural Rendering
• Lagrangian Neural Style Transfer for Fluids
• Correction by Projection: Denoising Images with Generative Adversarial Networks
• An Improved Evaluation Framework for Generative Adversarial Networks
• Generative Adversarial Networks (GANs): An Overview of Theoretical Model, Evaluation Metrics, and Recent Developments
• A Shape Transformation-based Dataset Augmentation Framework for Pedestrian Detection
• On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition
• AdaShift: Decorrelation and Convergence of Adaptive Learning Rate Methods
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• Regularization Methods for Generative Adversarial Networks: An Overview of Recent Studies
• SINR: Deconvolving Circular SAS Images Using Implicit Neural Representations
• HandAugment: A Simple Data Augmentation Method for Depth-Based 3D Hand Pose Estimation
• Unsupervised Learning of Shape and Pose with Differentiable Point Clouds
• SimNet: Enabling Robust Unknown Object Manipulation from Pure Synthetic Data via Stereo
• RTMV: A Ray-Traced Multi-View Synthetic Dataset for Novel View Synthesis
• ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations
• UncertaINR: Uncertainty Quantification of End-to-End Implicit Neural Representations for Computed Tomography
• Data Augmentation for Object Detection via Differentiable Neural Rendering
• Neural Photometry-guided Visual Attribute Transfer
• Automatic Generation of Machine Learning Synthetic Data Using ROS
• Random Shadows and Highlights: A new data augmentation method for extreme lighting conditions
• Hallucinated Adversarial Learning for Robust Visual Tracking
• Cross-Domain Image Classification through Neural-Style Transfer Data Augmentation
• Controllable Data Augmentation Through Deep Relighting

Cited by in corpus (2)

• Partitioned Neural Network Training via Synthetic Intermediate Labels
• Efficient Reasoning Distillation: Small Video-Language Models via Synthetic CoT and Difficulty-Aware Fine-Tuning