Overton: A Data System for Monitoring and Improving Machine-Learned Products

Abstract

We describe a system called Overton, whose main design goal is to support engineers in building, monitoring, and improving production machine learning systems. Key challenges engineers face are monitoring fine-grained quality, diagnosing errors in sophisticated applications, and handling contradictory or incomplete supervision data. Overton automates the life cycle of model construction, deployment, and monitoring by providing a set of novel high-level, declarative abstractions. Overton's vision is to shift developers to these higher-level tasks instead of lower-level machine learning tasks. In fact, using Overton, engineers can build deep-learning-based applications without writing any code in frameworks like TensorFlow. For over a year, Overton has been used in production to support multiple applications in both near-real-time applications and back-of-house processing. In that time, Overton-based applications have answered billions of queries in multiple languages and processed trillions of records reducing errors 1.7-2.9 times versus production systems.

References in corpus (4)

• Neural Architecture Search: A Survey
• Snorkel: Rapid Training Data Creation with Weak Supervision
• AutoAugment: Learning Augmentation Policies from Data
• Learning Dependency Structures for Weak Supervision Models

Cited by in corpus (6)

• On the Opportunities and Risks of Foundation Models
• Slice-based Learning: A Programming Model for Residual Learning in Critical Data Slices
• Interactive Weak Supervision: Learning Useful Heuristics for Data Labeling
• Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
• Model Monitoring and Dynamic Model Selection in Travel Time-series Forecasting
• Ember: No-Code Context Enrichment via Similarity-Based Keyless Joins