A Scaling Law for Synthetic-to-Real Transfer: How Much Is Your Pre-training Effective?

Abstract

Synthetic-to-real transfer learning is a framework in which a synthetically generated dataset is used to pre-train a model to improve its performance on real vision tasks. The most significant advantage of using synthetic images is that the ground-truth labels are automatically available, enabling unlimited expansion of the data size without human cost. However, synthetic data may have a huge domain gap, in which case increasing the data size does not improve the performance. How can we know that? In this study, we derive a simple scaling law that predicts the performance from the amount of pre-training data. By estimating the parameters of the law, we can judge whether we should increase the data or change the setting of image synthesis. Further, we analyze the theory of transfer learning by considering learning dynamics and confirm that the derived generalization bound is consistent with our empirical findings. We empirically validated our scaling law on various experimental settings of benchmark tasks, model sizes, and complexities of synthetic images.

References in corpus (21)

• Overcoming catastrophic forgetting in neural networks
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Shortcut Learning in Deep Neural Networks
• Scaling Laws for Neural Language Models
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Deep Learning Scaling is Predictable, Empirically
• Rethinking Pre-training and Self-training
• What makes ImageNet good for transfer learning?
• Scaling Laws for Autoregressive Generative Modeling
• Explaining Neural Scaling Laws
• The Origins and Prevalence of Texture Bias in Convolutional Neural Networks
• The Benefit of Multitask Representation Learning
• On the Theory of Transfer Learning: The Importance of Task Diversity
• Few-Shot Learning via Learning the Representation, Provably
• Hypothesis Transfer Learning via Transformation Functions
• Applying Domain Randomization to Synthetic Data for Object Category Detection
• An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection
• Scaling Laws for Transfer
• A Neural Scaling Law from the Dimension of the Data Manifold
• Learning Curve Theory

Cited by in corpus (2)

• Machine Learning For Elliptic PDEs: Fast Rate Generalization Bound, Neural Scaling Law and Minimax Optimality
• Scalable Modular Synthetic Data Generation for Advancing Aerial Autonomy