Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
arXiv:1909.06349
Abstract
In real-world machine learning applications, data subsets correspond to especially critical outcomes: vulnerable cyclist detections are safety-critical in an autonomous driving task, and "question" sentences might be important to a dialogue agent's language understanding for product purposes. While machine learning models can achieve high quality performance on coarse-grained metrics like F1-score and overall accuracy, they may underperform on critical subsets---we define these as slices, the key abstraction in our approach. To address slice-level performance, practitioners often train separate "expert" models on slice subsets or use multi-task hard parameter sharing. We propose Slice-based Learning, a new programming model in which the slicing function (SF), a programming interface, specifies critical data subsets for which the model should commit additional capacity. Any model can leverage SFs to learn slice expert representations, which are combined with an attention mechanism to make slice-aware predictions. We show that our approach maintains a parameter-efficient representation while improving over baselines by up to 19.0 F1 on slices and 4.6 F1 overall on datasets spanning language understanding (e.g. SuperGLUE), computer vision, and production-scale industrial systems.
NeurIPS 2019
References in corpus (7)
- An Overview of Multi-Task Learning in Deep Neural Networks
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
- WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations
- Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale
- Overton: A Data System for Monitoring and Improving Machine-Learned Products
- Gated networks: an inventory
- Probing What Different NLP Tasks Teach Machines about Function Word Comprehension
Cited by in corpus (6)
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Discovering and Validating AI Errors With Crowdsourced Failure Reports
- Zeno: An Interactive Framework for Behavioral Evaluation of Machine Learning
- ScatterShot: Interactive In-context Example Curation for Text Transformation
- Enabling Collaborative Data Science Development with the Ballet Framework
- VLSlice: Interactive Vision-and-Language Slice Discovery