ScatterShot: Interactive In-context Example Curation for Text Transformation
arXiv:2302.07346 · doi:10.1145/3581641.3584059
Abstract
The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.
IUI 2023: 28th International Conference on Intelligent User Interfaces
References in corpus (9)
- Snorkel: Rapid Training Data Creation with Weak Supervision
- Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
- CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
- Deep Active Learning over the Long Tail
- Selective Annotation Makes Language Models Better Few-Shot Learners
- Batch Active Learning at Scale
- Domino: Discovering Systematic Errors with Cross-Modal Embeddings
- ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
- Active Learning Helps Pretrained Models Learn the Intended Task