Adversarial Validation Approach to Concept Drift Problem in User Targeting Automation Systems at Uber

Abstract

In user targeting automation systems, concept drift in input data is one of the main challenges. It deteriorates model performance on new data over time. Previous research on concept drift mostly proposed model retraining after observing performance decreases. However, this approach is suboptimal because the system fixes the problem only after suffering from poor performance on new data. Here, we introduce an adversarial validation approach to concept drift problems in user targeting automation systems. With our approach, the system detects concept drift in new data before making inference, trains a model, and produces predictions adapted to the new data. We show that our approach addresses concept drift effectively with the AutoML3 Lifelong Machine Learning challenge data as well as in Uber's internal user targeting automation system, MaLTA.

arXiv comment12 pages, 4 figures

References in corpus (3)

• Scikit-learn: Machine Learning in Python
• Meta-learners for Estimating Heterogeneous Treatment Effects using Machine Learning
• Towards AutoML in the presence of Drift: first results

Cited by in corpus (3)

• Machine learning technique for morphological classification of galaxies from SDSS. II. The image-based morphological catalogs of galaxies at 0.02<z<0.1
• On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction
• Generalizing Fault Detection Against Domain Shifts Using Stratification-Aware Cross-Validation