Ranking and combining multiple predictors without labeled data

Abstract

In a broad range of classification and decision making problems, one is given the advice or predictions of several classifiers, of unknown reliability, over multiple questions or queries. This scenario is different from the standard supervised setting, where each classifier accuracy can be assessed using available labeled data, and raises two questions: given only the predictions of several classifiers over a large set of unlabeled test data, is it possible to a) reliably rank them; and b) construct a meta-classifier more accurate than most classifiers in the ensemble? Here we present a novel spectral approach to address these questions. First, assuming conditional independence between classifiers, we show that the off-diagonal entries of their covariance matrix correspond to a rank-one matrix. Moreover, the classifiers can be ranked using the leading eigenvector of this covariance matrix, as its entries are proportional to their balanced accuracies. Second, via a linear approximation to the maximum likelihood estimator, we derive the Spectral Meta-Learner (SML), a novel ensemble classifier whose weights are equal to this eigenvector entries. On both simulated and real data, SML typically achieves a higher accuracy than most classifiers in the ensemble and can provide a better starting point than majority voting, for estimating the maximum likelihood solution. Furthermore, SML is robust to the presence of small malicious groups of classifiers designed to veer the ensemble prediction away from the (unknown) ground truth.

Supplementary Information is included at the end of the manuscript. This is a revision of our original submission of the manuscript entitled "The student's dilemma: ranking and improving prediction at test time without access to training data", which is now entitled "Ranking and combining multiple predictors without labeled data"

Cited by in corpus (27)

• Snorkel: Rapid Training Data Creation with Weak Supervision
• Data Programming: Creating Large Training Sets, Quickly
• Learning the Structure of Generative Models without Labeled Data
• Spectral Methods meet EM: A Provably Optimal Algorithm for Crowdsourcing
• A Permutation-based Model for Crowd Labeling: Optimal Estimation and Robustness
• Blind Multiclass Ensemble Classification
• A Deep Learning Approach to Unsupervised Ensemble Learning
• Exact Exponent in Optimal Rates for Crowdsourcing
• Estimating the Accuracies of Multiple Classifiers Without Labeled Data
• T-TIME: Test-Time Information Maximization Ensemble for Plug-and-Play BCIs
• Analysis of Minimax Error Rate for Crowdsourcing and Its Application to Worker Clustering Model
• Transfer Learning for EEG-Based Brain-Computer Interfaces: A Review of Progress Made Since 2016
• Robust angle-based transfer learning in high dimensions
• Scalable Semi-Supervised Aggregation of Classifiers
• Wisdom of the crowd from unsupervised dimension reduction
• Unsupervised Evaluation and Weighted Aggregation of Ranked Predictions
• Unsupervised Ensemble Learning with Dependent Classifiers
• Aggregating Dependent Gaussian Experts in Local Approximation
• Learning from Imperfect Annotations
• Random Sampling in an Age of Automation: Minimizing Expenditures through Balanced Collection and Annotation
• A Streaming Algorithm for Crowdsourced Data Classification
• Copula Quadrant Similarity for Anomaly Scores
• Agreement Rate Initialized Maximum Likelihood Estimator for Ensemble Classifier Aggregation and Its Application in Brain-Computer Interface
• Blind Exploration and Exploitation of Stochastic Experts
• Agreement-based Learning
• StackingNet: Collective Inference Across Independent AI Foundation Models
• Active Semi-supervised Transfer Learning (ASTL) for Offline BCI Calibration