7 papers
Metadata Management for AI-Augmented Data Workflows
Jinjin Zhao, Sanjay Krishnan
AI-augmented data workflows introduce complex governance challenges, as both human and model-driven processes generate, transform, and consume data artifacts. These workflows blend…
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
Jinjin Zhao, Sanjay Krishnan
Large Language Models (LLMs) have emerged as powerful tools for automating and executing complex data tasks. However, their integration into more complex data workflows introduces…
Fast Capture of Cell-Level Provenance in Numpy
Jinjin Zhao, Sanjay Krishnan
Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows. However, significant challenges arise in capturing this provenance, includi…
A System for Quantifying Data Science Workflows with Fine-Grained Procedural Logging and a Pilot Study
Jinjin Zhao, Avidgor Gal, Sanjay Krishnan
It is important for researchers to understand precisely how data scientists turn raw data into insights, including typical programming patterns, workflow, and methodology. This pap…
Compression and In-Situ Query Processing for Fine-Grained Array Lineage
Jinjin Zhao, Sanjay Krishnan
Tracking data lineage is important for data integrity, reproducibility, and debugging data science workflows. However, fine-grained lineage (i.e., at a cell level) is challenging t…
Data Makes Better Data Scientists
Jinjin Zhao, Avidgor Gal, Sanjay Krishnan
With the goal of identifying common practices in data science projects, this paper proposes a framework for logging and understanding incremental code executions in Jupyter noteboo…