23 citations · 37 across the 5 of their papers we have counts for
5 papers
Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
Shreya Shankar, J. D. Zamfirescu-Pereira, Björn Hartmann +2
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating L…
"We Have No Idea How Models will Behave in Production until Production": How Engineers Operationalize Machine Learning
Shreya Shankar, Rolando Garcia, Joseph M Hellerstein +1
Organizations rely on machine learning engineers (MLEs) to deploy models and maintain ML pipelines in production. Due to models' extensive reliance on fresh data, the operationaliz…
SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
Shreya Shankar, Haotian Li, Parth Asawa +7
Large language models (LLMs) are being increasingly deployed as part of pipelines that repeatedly process or generate data of some sort. However, a common barrier to deployment are…
Revisiting Prompt Engineering via Declarative Crowdsourcing
Aditya G. Parameswaran, Shreya Shankar, Parth Asawa +2
Large language models (LLMs) are incredibly powerful at comprehending and generating data in the form of text, but are brittle and error-prone. There has been an advent of toolkits…
Moving Fast With Broken Data
Shreya Shankar, Labib Fawaz, Karl Gyllstrom +1
Machine learning (ML) models in production pipelines are frequently retrained on the latest partitions of large, continually-growing datasets. Due to engineering bugs, partitions i…