Automated Synthetic-to-Real Generalization
arXiv:2007.06965
Abstract
Models trained on synthetic images often face degraded generalization to real data. As a convention, these models are often initialized with ImageNet pre-trained representation. Yet the role of ImageNet knowledge is seldom discussed despite common practices that leverage this knowledge to maintain the generalization ability. An example is the careful hand-tuning of early stopping and layer-wise learning rates, which is shown to improve synthetic-to-real generalization but is also laborious and heuristic. In this work, we explicitly encourage the synthetically trained model to maintain similar representations with the ImageNet pre-trained model, and propose a \textit{learning-to-optimize (L2O)} strategy to automate the selection of layer-wise learning rates. We demonstrate that the proposed framework can significantly improve the synthetic-to-real generalization performance without seeing and training on real data, while also benefiting downstream tasks such as domain adaptation. Code is available at: https://github.com/NVlabs/ASG.
Accepted to ICML 2020
Cited by in corpus (5)
- Generalize then Adapt: Source-Free Domain Adaptive Semantic Segmentation
- Informing Geometric Deep Learning with Electronic Interactions to Accelerate Quantum Chemistry
- Learning to Optimize: A Primer and A Benchmark
- Contrastive Syn-to-Real Generalization
- WEDGE: Web-Image Assisted Domain Generalization for Semantic Segmentation