Progressive Data Science: Potential and Challenges

Abstract

Data science requires time-consuming iterative manual activities. In particular, activities such as data selection, preprocessing, transformation, and mining, highly depend on iterative trial-and-error processes that could be sped-up significantly by providing quick feedback on the impact of changes. The idea of progressive data science is to compute the results of changes in a progressive manner, returning a first approximation of results quickly and allow iterative refinements until converging to a final result. Enabling the user to interact with the intermediate results allows an early detection of erroneous or suboptimal choices, the guided definition of modifications to the pipeline and their quick assessment. In this paper, we discuss the progressiveness challenges arising in different steps of the data science pipeline. We describe how changes in each step of the pipeline impact the subsequent steps and outline why progressive data science will help to make the process more effective. Computing progressive approximations of outcomes resulting from changes creates numerous research challenges, especially if the changes are made in the early steps of the pipeline. We discuss these challenges and outline first steps towards progressiveness, which, we argue, will ultimately help to significantly speed-up the overall data science process.

References in corpus (5)

• Generative Adversarial Active Learning
• Progressive Analytics: A Computation Paradigm for Exploratory Data Analysis
• A Survey on Deep Learning Toolkits and Libraries for Intelligent User Interfaces
• An Interactive Machine Learning Framework
• Learning An Invariant Speech Representation

Cited by in corpus (7)

• An Incremental Dimensionality Reduction Method for Visualizing Streaming Multidimensional Data
• The Art and Practice of Data Science Pipelines: A Comprehensive Study of Data Science Pipelines In Theory, In-The-Small, and In-The-Large
• FeatureEnVi: Visual Analytics for Feature Engineering Using Stepwise Selection and Semi-Automatic Extraction Approaches
• VisEvol: Visual Analytics to Support Hyperparameter Search through Evolutionary Optimization
• DBPal: Weak Supervision for Learning a Natural Language Interface to Databases
• Towards Better Modeling with Missing Data: A Contrastive Learning-based Visual Analytics Perspective
• A Visual Analytics Framework for Reviewing Streaming Performance Data