How Much Automation Does a Data Scientist Want?

Abstract

Data science and machine learning (DS/ML) are at the heart of the recent advancements of many Artificial Intelligence (AI) applications. There is an active research thread in AI, \autoai, that aims to develop systems for automating end-to-end the DS/ML Lifecycle. However, do DS and ML workers really want to automate their DS/ML workflow? To answer this question, we first synthesize a human-centered AutoML framework with 6 User Role/Personas, 10 Stages and 43 Sub-Tasks, 5 Levels of Automation, and 5 Types of Explanation, through reviewing research literature and marketing reports. Secondly, we use the framework to guide the design of an online survey study with 217 DS/ML workers who had varying degrees of experience, and different user roles "matching" to our 6 roles/personas. We found that different user personas participated in distinct stages of the lifecycle -- but not all stages. Their desired levels of automation and types of explanation for AutoML also varied significantly depending on the DS/ML stage and the user persona. Based on the survey results, we argue there is no rationale from user needs for complete automation of the end-to-end DS/ML lifecycle. We propose new next steps for user-controlled DS/ML automation.

References in corpus (16)

• Towards A Rigorous Science of Interpretable Machine Learning
• AMC: AutoML for Model Compression and Acceleration on Mobile Devices
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm
• One Explanation Does Not Fit All: A Toolkit and Taxonomy of AI Explainability Techniques
• Metrics for Explainable AI: Challenges and Prospects
• Probabilistic Matrix Factorization for Automated Machine Learning
• One button machine for automating feature engineering in relational databases
• AutoAIViz: Opening the Blackbox of Automated Artificial Intelligence with Conditional Parallel Coordinates
• HoloClean: Holistic Data Repairs with Probabilistic Inference
• Easy Hyperparameter Search Using Optunity
• AlphaClean: Automatic Generation of Data Cleaning Pipelines
• Feature Engineering for Predictive Modeling using Reinforcement Learning
• An ADMM Based Framework for AutoML Pipeline Configuration
• Towards Automating the AI Operations Lifecycle
• PANDA: Facilitating Usable AI Development
• How can AI Automate End-to-End Data Science?

Cited by in corpus (7)

• Exploring How Machine Learning Practitioners (Try To) Use Fairness Toolkits
• Documentation Matters: Human-Centered AI System to Assist Data Science Code Documentation in Computational Notebooks
• AutoDS: Towards Human-Centered Automation of Data Science
• Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Work
• Enabling Collaborative Data Science Development with the Ballet Framework
• Human-Centered AI for Data Science: A Systematic Approach
• HAConvGNN: Hierarchical Attention Based Convolutional Graph Neural Network for Code Documentation Generation in Jupyter Notebooks