AutoML in The Wild: Obstacles, Workarounds, and Expectations

Abstract

Automated machine learning (AutoML) is envisioned to make ML techniques accessible to ordinary users. Recent work has investigated the role of humans in enhancing AutoML functionality throughout a standard ML workflow. However, it is also critical to understand how users adopt existing AutoML solutions in complex, real-world settings from a holistic perspective. To fill this gap, this study conducted semi-structured interviews of AutoML users (N=19) focusing on understanding (1) the limitations of AutoML encountered by users in their real-world practices, (2) the strategies users adopt to cope with such limitations, and (3) how the limitations and workarounds impact their use of AutoML. Our findings reveal that users actively exercise user agency to overcome three major challenges arising from customizability, transparency, and privacy. Furthermore, users make cautious decisions about whether and how to apply AutoML on a case-by-case basis. Finally, we derive design implications for developing future AutoML solutions.

In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI'23), April 23-28, 2023, Hamburg, Germany

References in corpus (8)

• Questioning the AI: Informing Design Practices for Explainable AI User Experiences
• Human-AI Collaboration in Data Science: Exploring Data Scientists' Perceptions of Automated AI
• Trust in Data Science: Collaboration, Translation, and Accountability in Corporate Data Science Projects
• Trust in AutoML: Exploring Information Needs for Establishing Trust in Automated Machine Learning Systems
• AutoDS: Towards Human-Centered Automation of Data Science
• AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates
• Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
• Learning Backward Compatible Embeddings

Cited by in corpus (4)

• Generative AI in the Wild: Prospects, Challenges, and Strategies
• Assessing the Use of AutoML for Data-Driven Software Engineering
• Trustworthy, responsible, ethical AI in manufacturing and supply chains: synthesis and emerging research questions
• Practitioner Motives to Use Different Hyperparameter Optimization Methods