Rekall: Specifying Video Events using Compositions of Spatiotemporal Labels

Abstract

Many real-world video analysis applications require the ability to identify domain-specific events in video, such as interviews and commercials in TV news broadcasts, or action sequences in film. Unfortunately, pre-trained models to detect all the events of interest in video may not exist, and training new models from scratch can be costly and labor-intensive. In this paper, we explore the utility of specifying new events in video in a more traditional manner: by writing queries that compose outputs of existing, pre-trained models. To write these queries, we have developed Rekall, a library that exposes a data model and programming model for compositional video event specification. Rekall represents video annotations from different sources (object detectors, transcripts, etc.) as spatiotemporal labels associated with continuous volumes of spacetime in a video, and provides operators for composing labels into queries that model new video events. We demonstrate the use of Rekall in analyzing video from cable TV news broadcasts, films, static-camera vehicular video streams, and commercial autonomous vehicle logs. In these efforts, domain experts were able to quickly (in a few hours to a day) author queries that enabled the accurate detection of new events (on par with, and in some cases much more accurate than, learned approaches) and to rapidly retrieve video clips for human-in-the-loop tasks such as video content curation and training data curation. Finally, in a user study, novice users of Rekall were able to author queries to retrieve new events in video given just one hour of query development time.

References in corpus (8)

• Prototypical Networks for Few-shot Learning
• Zero-Shot Learning Through Cross-Modal Transfer
• Snorkel: Rapid Training Data Creation with Weak Supervision
• Scanner: Efficient Video Analysis at Scale
• Large-scale, Fast and Accurate Shot Boundary Detection through Spatio-temporal Convolutional Neural Networks
• Inferring Generative Model Structure with Static Analysis
• BlazeIt: Optimizing Declarative Aggregation and Limit Queries for Neural Network-Based Video Analytics
• Fine-Grained Retrieval of Sports Plays using Tree-Based Alignment of Trajectories

Cited by in corpus (10)

• WRENCH: A Comprehensive Benchmark for Weak Supervision
• Fast and Three-rious: Speeding Up Weak Supervision with Triplet Methods
• Train and You'll Miss It: Interactive Model Iteration with Weak Supervision and Pre-Trained Embeddings
• Multi-Resolution Weak Supervision for Sequential Data
• Analyzing Who and What Appears in a Decade of US Cable TV News
• Universal Embeddings for Spatio-Temporal Tagging of Self-Driving Logs
• Top-K Deep Video Analytics: A Probabilistic Approach
• Investigating naturalistic hand movements by behavior mining in long-term video and neural recordings
• Efficient Pipelines for Vision-Based Context Sensing
• A Survey of Performance Optimization in Neural Network-Based Video Analytics Systems