155 citations · 251 across the 23 of their papers we have counts for
26 papers
Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune
Bhavya Chopra, Meng Chen, Rebecca Dang +5
Large language models (LLMs) are increasingly used to score text records at scale (e.g., rating candidate resumes on a 1-5 scale). However, existing LLM-powered approaches do not a…
Scout: Scalable Document Extraction via Data Similarity
Yiming Lin, Chiyu Hao, Shreya Shankar +1
Extracting values from large document collections powers data analysis across many domains. Frontier LLMs extract such values accurately, but processing an entire collection with o…
Semantic Data Processing with Holistic Data Understanding
Youran Sun, Sepanta Zeighami, Bhavya Chopra +2
Semantic operators have increasingly become integrated within data systems to enable processing data using Large Language Models (LLMs). Despite significant recent effort in improv…
Can AI Agents Answer Your Data Questions? A Benchmark for Data Agents
Ruiying Ma, Shreya Shankar, Ruiqi Chen +7
However, building reliable data agents remains difficult because real enterprise data is fragmented across many heterogeneous database systems, with duplicated and inconsistent dat…
Task Cascades for Efficient Unstructured Data Processing
Shreya Shankar, Sepanta Zeighami, Aditya Parameswaran
Modern database systems allow users to query or process unstructured text or document columns using LLM-powered functions. Users can express an operation in natural language (e.g.,…
Featurized-Decomposition Join: Low-Cost Semantic Joins with Guarantees
Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran
Large Language Models (LLMs) are being increasingly used within data systems to process large datasets with text fields. A broad class of such tasks involves a semantic join-joinin…
Multi-Objective Agentic Rewrites for Unstructured Data Processing
Lindsey Linxi Wei, Shreya Shankar, Sepanta Zeighami +3
One year ago, we open-sourced DocETL, a declarative system for LLM-powered data processing that, as of March 2026, has 3.7K GitHub stars and users across domains (e.g., journalism,…
Cut Costs, Not Accuracy: LLM-Powered Data Processing with Guarantees
Sepanta Zeighami, Shreya Shankar, Aditya Parameswaran
Large Language Models (LLMs) are being increasingly used as a building block in data systems to process large text datasets. To do so, LLM model providers offer multiple LLMs with…
Supporting Our AI Overlords: Redesigning Data Systems to be Agent-First
Shu Liu, Soujanya Ponnapalli, Shreya Shankar +12
Large Language Model (LLM) agents, acting on their users' behalf to manipulate and analyze data, are likely to become the dominant workload for data systems in the future. When wor…
Rethinking Dataset Discovery with DataScout
Rachel Lin, Bhavya Chopra, Wenjing Lin +3
Dataset Search -- the process of finding appropriate datasets for a given task -- remains a critical yet under-explored challenge in data science workflows. Assessing dataset suita…
Steering Semantic Data Processing With DocWrangler
Shreya Shankar, Bhavya Chopra, Mawil Hasan +5
Unstructured text has long been difficult to automatically analyze at scale. Large language models (LLMs) now offer a way forward by enabling {\em semantic data processing}, where…
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom Production Large Language Model Pipelines
Reya Vir, Shreya Shankar, Harrison Chase +2
Large language models (LLMs) are increasingly deployed in specialized production data processing pipelines across diverse domains -- such as finance, marketing, and e-commerce. How…
RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines
Quentin Romero Lauro, Shreya Shankar, Sepanta Zeighami +1
Retrieval-augmented generation (RAG) pipelines have become the de-facto approach for building AI assistants with access to external, domain-specific knowledge. Given a user query,…
LLM-Powered Proactive Data Systems
Sepanta Zeighami, Yiming Lin, Shreya Shankar +1
With the power of LLMs, we now have the ability to query data that was previously impossible to query, including text, images, and video. However, despite this enormous potential,…
DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processing
Shreya Shankar, Tristan Chambers, Tarak Shah +2
Analyzing unstructured data has been a persistent challenge in data processing. Large Language Models (LLMs) have shown promise in this regard, leading to recent proposals for decl…
Towards Accurate and Efficient Document Analytics with Large Language Models
Yiming Lin, Madelon Hulsebos, Ruiying Ma +4
Unstructured data formats account for over 80% of the data currently stored, and extracting value from such formats remains a considerable challenge. In particular, current approac…
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shreya Shankar, J. D. Zamfirescu-Pereira, Björn Hartmann +2
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating L…
"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
Shreya Shankar, Rolando Garcia, Joseph M Hellerstein +1
Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationaliz…
SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
Shreya Shankar, Haotian Li, Parth Asawa +7
Large language models (LLMs) are being increasingly deployed as part of pipelines that repeatedly process or generate data of some sort. However, a common barrier to deployment are…
Revisiting Prompt Engineering via Declarative Crowdsourcing
Aditya G. Parameswaran, Shreya Shankar, Parth Asawa +2
Large language models (LLMs) are incredibly powerful at comprehending and generating data in the form of text, but are brittle and error-prone. There has been an advent of toolkits…
Moving Fast With Broken Data
Shreya Shankar, Labib Fawaz, Karl Gyllstrom +1
Machine learning (ML) models in production pipelines are frequently retrained on the latest partitions of large, continually-growing datasets. Due to engineering bugs, partitions i…
Operationalizing Machine Learning: An Interview Study
Shreya Shankar, Rolando Garcia, Joseph M. Hellerstein +1
Organizations rely on machine learning engineers (MLEs) to operationalize ML, i.e., deploy and maintain ML pipelines in production. The process of operationalizing ML, or MLOps, co…
Rethinking Streaming Machine Learning Evaluation
Shreya Shankar, Bernease Herman, Aditya G. Parameswaran
While most work on evaluating machine learning (ML) models focuses on computing accuracy on batches of data, tracking accuracy alone in a streaming setting (i.e., unbounded, timest…
Towards Observability for Production Machine Learning Pipelines
Shreya Shankar, Aditya Parameswaran
Software organizations are increasingly incorporating machine learning (ML) into their product offerings, driving a need for new data management tools. Many of these tools facilita…