Towards Out-Of-Distribution Generalization: A Survey

Abstract

Traditional machine learning paradigms are based on the assumption that both training and test data follow the same statistical pattern, which is mathematically referred to as Independent and Identically Distributed ($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often fails to hold due to unforeseen distributional shifts, leading to considerable degradation in model performance upon deployment. This observed discrepancy indicates the significance of investigating the Out-of-Distribution (OOD) generalization problem. OOD generalization is an emerging topic of machine learning research that focuses on complex scenarios wherein the distributions of the test data differ from those of the training data. This paper represents the first comprehensive, systematic review of OOD generalization, encompassing a spectrum of aspects from problem definition, methodological development, and evaluation procedures, to the implications and future directions of the field. Our discussion begins with a precise, formal characterization of the OOD generalization problem. Following that, we categorize existing methodologies into three segments: unsupervised representation learning, supervised model learning, and optimization, according to their positions within the overarching learning process. We provide an in-depth discussion on representative methodologies for each category, further elucidating the theoretical links between them. Subsequently, we outline the prevailing benchmark datasets employed in OOD generalization studies. To conclude, we overview the existing body of work in this domain and suggest potential avenues for future research on OOD generalization. A summary of the OOD generalization methodologies surveyed in this paper can be accessed at http://out-of-distribution-generalization.com.

51 pages

References in corpus (21)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Deep Domain Confusion: Maximizing for Domain Invariance
• Equality of Opportunity in Supervised Learning
• Predictive learning via rule ensembles
• Domain Generalization via Invariant Feature Representation
• Domain Generalization via Model-Agnostic Learning of Semantic Features
• Domain Adaptation for Visual Applications: A Comprehensive Survey
• Unsupervised Domain Adaptation through Self-Supervision
• Towards Causal Representation Learning
• Feature-Critic Networks for Heterogeneous Domain Generalization
• Frustratingly Simple Domain Generalization via Image Stylization
• Self-Challenging Improves Cross-Domain Generalization
• Towards a Theoretical Framework of Out-of-Distribution Generalization
• Repairing without Retraining: Avoiding Disparate Impact with Counterfactual Distributions
• Does Invariant Risk Minimization Capture Invariance?
• A Generalization Error Bound for Multi-class Domain Generalization
• Linear unit-tests for invariance discovery
• Out-of-Distribution Generalization Analysis via Influence Function
• Incorporating Unlabeled Data into Distributionally Robust Learning
• The iWildCam 2020 Competition Dataset
• Heterogeneous Risk Minimization

Cited by in corpus (30)

• An overview of artificial intelligence techniques for diagnosis of Schizophrenia based on magnetic resonance imaging modalities: Methods, challenges, and future works
• Self-supervised remote sensing feature learning: Learning Paradigms, Challenges, and Future Works
• Learning Disentangled Representations in the Imaging Domain
• Domain Generalization for Medical Image Analysis: A Review
• Deep Long-Tailed Learning: A Survey
• Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Review
• Alleviating Structural Distribution Shift in Graph Anomaly Detection
• Invariant Collaborative Filtering to Popularity Distribution Shift
• CausPref: Causal Preference Learning for Out-of-Distribution Recommendation
• Towards out of distribution generalization for problems in mechanics
• Perfect is the enemy of test oracle
• Reformulating CTR Prediction: Learning Invariant Feature Interactions for Recommendation
• Causal invariant geographic network representations with feature and structural distribution shifts
• Deep neural networks for choice analysis: Enhancing behavioral regularity with gradient regularization
• Finding emergence in data by maximizing effective information
• Fourier-basis Functions to Bridge Augmentation Gap: Rethinking Frequency Augmentation in Image Classification
• Debiasing Sequential Recommenders through Distributionally Robust Optimization over System Exposure
• PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation
• Emergency Department Decision Support using Clinical Pseudo-notes
• Regulatory Instruments for Fair Personalized Pricing
• An Offline Metric for the Debiasedness of Click Models
• A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision
• SAGE-HB: Swift Adaptation and Generalization in Massive MIMO Hybrid Beamforming
• Pretrained Embeddings for E-commerce Machine Learning: When it Fails and Why?
• Learning a quantum computer's capability
• DREAM: Domain-agnostic Reverse Engineering Attributes of Black-box Model
• Coverage-Guaranteed Prediction Sets for Out-of-Distribution Data
• Deep Out-of-Distribution Uncertainty Quantification via Weight Entropy Maximization
• Confounder Identification-free Causal Visual Feature Learning
• A benchmark with decomposed distribution shifts for 360 monocular depth estimation