155 citations · 251 across the 23 of their papers we have counts for
6 papers · 1 filter
Who's Keeping Score? Interactive Steering of LLM-Powered Scoring with Attune
Bhavya Chopra, Meng Chen, Rebecca Dang +5
Large language models (LLMs) are increasingly used to score text records at scale (e.g., rating candidate resumes on a 1-5 scale). However, existing LLM-powered approaches do not a…
Rethinking Dataset Discovery with DataScout
Rachel Lin, Bhavya Chopra, Wenjing Lin +3
Dataset Search -- the process of finding appropriate datasets for a given task -- remains a critical yet under-explored challenge in data science workflows. Assessing dataset suita…
Steering Semantic Data Processing With DocWrangler
Shreya Shankar, Bhavya Chopra, Mawil Hasan +5
Unstructured text has long been difficult to automatically analyze at scale. Large language models (LLMs) now offer a way forward by enabling {\em semantic data processing}, where…
RAG Without the Lag: Interactive Debugging for Retrieval-Augmented Generation Pipelines
Quentin Romero Lauro, Shreya Shankar, Sepanta Zeighami +1
Retrieval-augmented generation (RAG) pipelines have become the de-facto approach for building AI assistants with access to external, domain-specific knowledge. Given a user query,…
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shreya Shankar, J. D. Zamfirescu-Pereira, Björn Hartmann +2
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating L…
"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
Shreya Shankar, Rolando Garcia, Joseph M Hellerstein +1
Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationaliz…