8 papers
Metadata Management for AI-Augmented Data Workflows
Jinjin Zhao, Sanjay Krishnan
AI-augmented data workflows introduce complex governance challenges, as both human and model-driven processes generate, transform, and consume data artifacts. These workflows blend…
TableVault: Managing Dynamic Data Collections for LLM-Augmented Workflows
Jinjin Zhao, Sanjay Krishnan
Large Language Models (LLMs) have emerged as powerful tools for automating and executing complex data tasks. However, their integration into more complex data workflows introduces…
Fast Capture of Cell-Level Provenance in Numpy
Jinjin Zhao, Sanjay Krishnan
Effective provenance tracking enhances reproducibility, governance, and data quality in array workflows. However, significant challenges arise in capturing this provenance, includi…
A System for Quantifying Data Science Workflows with Fine-Grained Procedural Logging and a Pilot Study
Jinjin Zhao, Avidgor Gal, Sanjay Krishnan
It is important for researchers to understand precisely how data scientists turn raw data into insights, including typical programming patterns, workflow, and methodology. This pap…
Compression and In-Situ Query Processing for Fine-Grained Array Lineage
Jinjin Zhao, Sanjay Krishnan
Tracking data lineage is important for data integrity, reproducibility, and debugging data science workflows. However, fine-grained lineage (i.e., at a cell level) is challenging t…
Data Makes Better Data Scientists
Jinjin Zhao, Avidgor Gal, Sanjay Krishnan
With the goal of identifying common practices in data science projects, this paper proposes a framework for logging and understanding incremental code executions in Jupyter noteboo…
Towards Causal Physical Error Discovery in Video Analytics Systems
Jinjin Zhao, Ted Shaowang, Stavos Sintos +1
Video analytics systems based on deep learning models are often opaque and brittle and require explanation systems to help users debug. Current model explanation system are very go…
Data Station: Delegated, Trustworthy, and Auditable Computation to Enable Data-Sharing Consortia with a Data Escrow
Siyuan Xia, Zhiru Zhu, Chris Zhu +7
Pooling and sharing data increases and distributes its value. But since data cannot be revoked once shared, scenarios that require controlled release of data for regulatory, privac…