Retrain or not retrain: Conformal test martingales for change-point detection

Abstract

We argue for supplementing the process of training a prediction algorithm by setting up a scheme for detecting the moment when the distribution of the data changes and the algorithm needs to be retrained. Our proposed schemes are based on exchangeability martingales, i.e., processes that are martingales under any exchangeable distribution for the data. Our method, based on conformal prediction, is general and can be applied on top of any modern prediction algorithm. Its validity is guaranteed, and in this paper we make first steps in exploring its efficiency.

arXiv comment22 pages, 19 figures, 3 tables

References in corpus (2)

• Scikit-learn: Machine Learning in Python
• Testing randomness

Cited by in corpus (8)

• Testing for Outliers with Conformal p-values
• A Two-Sample Conditional Distribution Test Using Conformal Prediction and Weighted Rank Sum
• Conformal testing in a binary model situation
• Tracking the risk of a deployed model and detecting harmful distribution shifts
• Conformal inference for random objects
• A unified framework for bandit multiple testing
• Enhancement of prediction algorithms by betting
• Testing exchangeability: fork-convexity, supermartingales, and e-processes