FunnelAL: Retrieve-then-Rank Active Learning for Single-Class Discovery

Abstract

We present FunnelAL, a retrieve-then-rank active learning system for single-class discovery, which adapts the multi-stage funnel architecture of industrial recommender systems to data annotation. Starting from one positive and one negative example, FunnelAL repeats three stages: (1) embedding-based retrieval that narrows the corpus to a candidate set, (2) a ranking stage that exploits a learned ranker (RankNet) and, once batch precision falls, mixes in committee-based exploration (QBC), and (3) the annotator's labels, which refine both stages for the next round. We evaluate on five datasets, three benchmarks and two large corpora of up to 2.7 million images, each with two different pretrained encoders, DINOv2 and CLIP. With clean labels and DINOv2, FunnelAL has the best mean rank over the five datasets, and with CLIP it shares the best rank with its ablation Funnel-RankNet. On the three benchmarks it discovers positives significantly faster than the classical and uncertainty-based baselines and than PF-MA, and no significant difference from PF-MA's final F1 was detected. With DINOv2, no significant difference from SEALS's active-search mode (most-likely-positive selection) was detected on most datasets. With CLIP, FunnelAL discovers significantly faster than it on three of the five. On iNaturalist's working set of 2.15 million images FunnelAL selects eight times faster than the neighborhood baselines as we implemented them (SEALS in both its modes and AnchorAL, which select only among the nearest neighbors of labeled items). There, PF-MA and uncertainty sampling, which label items near the decision boundary across the whole pool, reach a higher F1 under our evaluation protocol. Under 10% annotator label noise it has a higher F1 than every baseline on three of the five datasets with DINOv2 and on all five with CLIP, and SEALS's active-search mode is more robust at higher rates.

arXiv comment36 pages, 5 figures. v2 corrects an ablation implementation error and updates SEALS and AnchorAL to better align with their published algorithms. These changes affect some reported comparisons and narrow our conclusions about the benefit of exploration. Adds iNaturalist, fMoW and CLIP evaluations