Specification Overfitting in Artificial Intelligence

Abstract

Machine learning (ML) and artificial intelligence (AI) approaches are often criticized for their inherent bias and for their lack of control, accountability, and transparency. Consequently, regulatory bodies struggle with containing this technology's potential negative side effects. High-level requirements such as fairness and robustness need to be formalized into concrete specification metrics, imperfect proxies that capture isolated aspects of the underlying requirements. Given possible trade-offs between different metrics and their vulnerability to over-optimization, integrating specification metrics in system development processes is not trivial. This paper defines specification overfitting, a scenario where systems focus excessively on specified metrics to the detriment of high-level requirements and task performance. We present an extensive literature survey to categorize how researchers propose, measure, and optimize specification metrics in several AI fields (e.g., natural language processing, computer vision, reinforcement learning). Using a keyword-based search on papers from major AI conferences and journals between 2018 and mid-2023, we identify and analyze 74 papers that propose or optimize specification metrics. We find that although most papers implicitly address specification overfitting (e.g., by reporting more than one specification metric), they rarely discuss which role specification metrics should play in system development or explicitly define the scope and assumptions behind metric formulations.

41 pages, 2 figures. This version of the article has been accepted for publication, after peer review but is not the Version of Record and does not reflect post-acceptance improvements, or any corrections. The Version of Record is available online at: https://doi.org/10.1007/s10462-024-11040-6

References in corpus (14)

• A Comprehensive Survey on Graph Neural Networks
• Artificial Intelligence: the global landscape of ethics guidelines
• Survey of Hallucination in Natural Language Generation
• A Brief Survey of Deep Reinforcement Learning
• Geometric deep learning: going beyond Euclidean data
• The Ethics of AI Ethics -- An Evaluation of Guidelines
• Deep Neural Networks and Tabular Data: A Survey
• Demystifying the Draft EU Artificial Intelligence Act
• Adversarial Attack and Defense on Graph Data: A Survey
• Measurement and Fairness
• Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved
• FlipTest: Fairness Testing via Optimal Transport
• Fairness in Risk Assessment Instruments: Post-Processing to Achieve Counterfactual Equalized Odds
• Cross-functional Analysis of Generalisation in Behavioural Learning