Regression Bugs Are In Your Model! Measuring, Reducing and Analyzing Regressions In NLP Model Updates

Abstract

Behavior of deep neural networks can be inconsistent between different versions. Regressions during model update are a common cause of concern that often over-weigh the benefits in accuracy or efficiency gain. This work focuses on quantifying, reducing and analyzing regression errors in the NLP model updates. Using negative flip rate as regression measure, we show that regression has a prevalent presence across tasks in the GLUE benchmark. We formulate the regression-free model updates into a constrained optimization problem, and further reduce it into a relaxed form which can be approximately optimized through knowledge distillation training method. We empirically analyze how model ensemble reduces regression. Finally, we conduct CheckList behavioral testing to understand the distribution of regressions across linguistic phenomena, and the efficacy of ensemble and distillation methods.

arXiv comment13 pages, 3 figures, Accepted at ACL 2021 main conference

References in corpus (10)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Distilling the Knowledge in a Neural Network
• Language Models are Few-Shot Learners
• A continual learning survey: Defying forgetting in classification tasks
• Cross-lingual Language Model Pretraining
• Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• LAMOL: LAnguage MOdeling for Lifelong Language Learning
• Lifelong Language Knowledge Distillation
• Positive-Congruent Training: Towards Regression-Free Model Updates