ScatterShot: Interactive In-context Example Curation for Text Transformation

Abstract

The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions that fall short on unseen cases. Further, it is hard to know when "enough" examples have been included even for known patterns. In this work, we present ScatterShot, an interactive system for building high-quality demonstration sets for in-context learning. ScatterShot iteratively slices unlabeled data into task-specific patterns, samples informative inputs from underexplored or not-yet-saturated slices in an active learning manner, and helps users label more efficiently with the help of an LLM and the current example set. In simulation studies on two text perturbation scenarios, ScatterShot sampling improves the resulting few-shot functions by 4-5 percentage points over random sampling, with less variance as more examples are added. In a user study, ScatterShot greatly helps users in covering different patterns in the input space and labeling in-context examples more efficiently, resulting in better in-context learning and less user effort.

arXiv commentIUI 2023: 28th International Conference on Intelligent User Interfaces

References in corpus (16)

• Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Active Learning for Convolutional Neural Networks: A Core-Set Approach
• Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing
• Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization
• CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities
• Deep Active Learning over the Long Tail
• Selective Annotation Makes Language Models Better Few-Shot Learners
• Batch Active Learning at Scale
• Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension
• Domino: Discovering Systematic Errors with Cross-Modal Embeddings
• Deep Ensemble Bayesian Active Learning : Addressing the Mode Collapse issue in Monte Carlo dropout via Ensembles
• Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
• Interpretable Directed Diversity: Leveraging Model Explanations for Iterative Crowd Ideation
• ICE: Enabling Non-Experts to Build Models Interactively for Large-Scale Lopsided Problems
• Active Learning Helps Pretrained Models Learn the Intended Task

Cited by in corpus (5)

• EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteria
• A Taxonomy for Human-LLM Interaction Modes: An Initial Exploration
• A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness
• Exploring the Role of Social Support when Integrating Generative AI into Small Business Workflows
• OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Models