Publications (24)
Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
Vincent S. Chen, Sen Wu, Zhenzhen Weng +2
In real-world machine learning applications, data subsets correspond to especially critical outcomes: vulnerable cyclist detections are safety-critical in an autonomous driving tas…
Snorkel: Rapid Training Data Creation with Weak Supervision
Alexander Ratner, Stephen H. Bach, Henry Ehrenberg +3
Labeling training data is increasingly the largest bottleneck in deploying machine learning systems. We present Snorkel, a first-of-its-kind system that enables users to train stat…
Learning Dependency Structures for Weak Supervision Models
Paroma Varma, Frederic Sala, Ann He +2
Labeling training data is a key bottleneck in the modern machine learning pipeline. Recent weak supervision approaches combine labels from multiple noisy sources by estimating thei…
Towards Acyclic Preference Evaluation of Language Models via Multiple Evaluators
Zhengyu Hu, Jieyu Zhang, Zhihan Xiong +3
Despite the remarkable success of Large Language Models (LLMs), evaluating their outputs' quality regarding preference remains a critical challenge. While existing works usually le…
Nemo: Guiding and Contextualizing Weak Supervision for Interactive Data Programming
Cheng-Yu Hsieh, Jieyu Zhang, Alexander Ratner
Weak Supervision (WS) techniques allow users to efficiently create large training datasets by programmatically labeling data with heuristic sources of supervision. While the succes…
Training Complex Models with Multi-Task Weak Supervision
Alexander Ratner, Braden Hancock, Jared Dunnmon +3
As machine learning models continue to increase in complexity, collecting large hand-labeled training sets has become one of the biggest roadblocks in practice. Instead, weaker for…
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
Stephen H. Bach, Daniel Rodriguez, Yintao Liu +10
Labeling training data is one of the most costly bottlenecks in developing machine learning-based applications. We present a first-of-its-kind study showing how existing knowledge…
Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias
Yue Yu, Yuchen Zhuang, Jieyu Zhang +5
Large language models (LLMs) have been recently leveraged as training data generators for various natural language processing (NLP) tasks. While previous research has explored diff…
Binary Classification with Positive Labeling Sources
Jieyu Zhang, Yujing Wang, Yaming Yang +2
To create a large amount of training labels for machine learning models effectively and efficiently, researchers have turned to Weak Supervision (WS), which uses programmatic label…
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes
Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh +6
Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller t…
Learning the Structure of Generative Models without Labeled Data
Stephen H. Bach, Bryan He, Alexander Ratner +1
Curating labeled training data has become the primary bottleneck in machine learning. Recent frameworks address this bottleneck with generative models to synthesize labels at scale…
Cross-Modal Data Programming Enables Rapid Medical Machine Learning
Jared Dunnmon, Alexander Ratner, Nishith Khandwala +8
Labeling training datasets has become a key barrier to building medical machine learning models. One strategy is to generate training labels programmatically, for example by applyi…
A Survey on Programmatic Weak Supervision
Jieyu Zhang, Cheng-Yu Hsieh, Yue Yu +2
Labeling training data has become one of the major roadblocks to using machine learning. Among various weak supervision paradigms, programmatic weak supervision (PWS) has achieved…
Data Programming: Creating Large Training Sets, Quickly
Alexander Ratner, Christopher De Sa, Sen Wu +2
Large labeled training sets are the critical building blocks of supervised learning methods and are key enablers of deep learning techniques. For some applications, creating labele…
Leveraging Instance Features for Label Aggregation in Programmatic Weak Supervision
Jieyu Zhang, Linxin Song, Alexander Ratner
Programmatic Weak Supervision (PWS) has emerged as a widespread paradigm to synthesize training labels efficiently. The core component of PWS is the label model, which infers true…
MaskSearch: Querying Image Masks at Scale
Dong He, Jieyu Zhang, Maureen Daum +2
Machine learning tasks over image databases often generate masks that annotate image content (e.g., saliency maps, segmentation maps, depth maps) and enable a variety of applicatio…
On the Trade-off of Intra-/Inter-class Diversity for Supervised Pre-training
Jieyu Zhang, Bohan Wang, Zhengyu Hu +2
Pre-training datasets are critical for building state-of-the-art machine learning models, motivating rigorous study on their impact on downstream tasks. In this work, we study the…
DataComp: In search of the next generation of multimodal datasets
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang +31
Multimodal datasets are a critical component in recent breakthroughs such as Stable Diffusion and GPT-4, yet their design does not receive the same research attention as model arch…
Tool Documentation Enables Zero-Shot Tool-Usage with Large Language Models
Cheng-Yu Hsieh, Si-An Chen, Chun-Liang Li +5
Today, large language models (LLMs) are taught to use new tools by providing a few demonstrations of the tool's usage. Unfortunately, demonstrations are hard to acquire, and can re…
Creating Training Sets via Weak Indirect Supervision
Jieyu Zhang, Bohan Wang, Xiangchen Song +4
Creating labeled training sets has become one of the major roadblocks in machine learning. To address this, recent \emph{Weak Supervision (WS)} frameworks synthesize training label…
Understanding Programmatic Weak Supervision via Source-aware Influence Function
Jieyu Zhang, Haonan Wang, Cheng-Yu Hsieh +1
Programmatic Weak Supervision (PWS) aggregates the source votes of multiple weak supervision sources into probabilistic training labels, which are in turn used to train an end mode…
WRENCH: A Comprehensive Benchmark for Weak Supervision
Jieyu Zhang, Yue Yu, Yinghao Li +4
Recent Weak Supervision (WS) approaches have had widespread success in easing the bottleneck of labeling training data for machine learning by synthesizing labels from multiple pot…
MLSys: The New Frontier of Machine Learning Systems
Alexander Ratner, Dan Alistarh, Gustavo Alonso +66
Machine learning (ML) techniques are enjoying rapidly increasing adoption. However, designing and implementing the systems that support ML models in real-world deployments remains…
Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li +8
Large language models (LLMs), even when specifically trained to process long input contexts, struggle to capture relevant information located in the middle of their input. This phe…