The BOSARIS Toolkit: Theory, Algorithms and Code for Surviving the New DCF

Abstract

The change of two orders of magnitude in the 'new DCF' of NIST's SRE'10, relative to the 'old DCF' evaluation criterion, posed a difficult challenge for participants and evaluator alike. Initially, participants were at a loss as to how to calibrate their systems, while the evaluator underestimated the required number of evaluation trials. After the fact, it is now obvious that both calibration and evaluation require very large sets of trials. This poses the challenges of (i) how to decide what number of trials is enough, and (ii) how to process such large data sets with reasonable memory and CPU requirements. After SRE'10, at the BOSARIS Workshop, we built solutions to these problems into the freely available BOSARIS Toolkit. This paper explains the principles and algorithms behind this toolkit. The main contributions of the toolkit are: 1. The Normalized Bayes Error-Rate Plot, which analyses likelihood- ratio calibration over a wide range of DCF operating points. These plots also help in judging the adequacy of the sizes of calibration and evaluation databases. 2. Efficient algorithms to compute DCF and minDCF for large score files, over the range of operating points required by these plots. 3. A new score file format, which facilitates working with very large trial lists. 4. A faster logistic regression optimizer for fusion and calibration. 5. A principled way to define EER (equal error rate), which is of practical interest when the absolute error count is small.

presented at: The NIST SRE'11 Analysis Workshop, Atlanta, December 2011

Cited by in corpus (36)

• Integrating Frequency Translational Invariance in TDNNs and Frequency Positional Information in 2D ResNets to Enhance Speaker Verification
• UR Channel-Robust Synthetic Speech Detection System for ASVspoof 2021
• The IDLAB VoxSRC-20 Submission: Large Margin Fine-Tuning and Quality-Aware Score Calibration in DNN Based Speaker Verification
• The DKU-DukeECE Systems for VoxCeleb Speaker Recognition Challenge 2020
• Cross-Lingual Speaker Verification with Domain-Balanced Hard Prototype Mining and Language-Dependent Score Normalization
• ASSERT: Anti-Spoofing with Squeeze-Excitation and Residual neTworks
• A Study On Convolutional Neural Network Based End-To-End Replay Anti-Spoofing
• The xx205 System for the VoxCeleb Speaker Recognition Challenge 2020
• HLT-NUS SUBMISSION FOR 2020 NIST Conversational Telephone Speech SRE
• A comparison of linear and non-linear calibrations for speaker recognition
• Tackling the Score Shift in Cross-Lingual Speaker Verification by Exploiting Language Information
• LMD: A Learnable Mask Network to Detect Adversarial Examples for Speaker Verification
• Deepfake Detection System for the ADD Challenge Track 3.2 Based on Score Fusion
• Classification of Speech with and without Face Mask using Acoustic Features
• Beijing ZKJ-NPU Speaker Verification System for VoxCeleb Speaker Recognition Challenge 2021
• KU-ISPL Speaker Recognition Systems under Language mismatch condition for NIST 2016 Speaker Recognition Evaluation
• Spoofing Attack Detection using the Non-linear Fusion of Sub-band Classifiers
• An explainability study of the constant Q cepstral coefficient spoofing countermeasure for automatic speaker verification
• Optimizing Multi-Taper Features for Deep Speaker Verification
• Triplet Based Embedding Distance and Similarity Learning for Text-independent Speaker Verification
• t-EER: Parameter-Free Tandem Evaluation of Countermeasures and Biometric Comparators
• Learnable MFCCs for Speaker Verification
• End-to-end anti-spoofing with RawNet2
• LEAP System for SRE19 CTS Challenge -- Improvements and Error Analysis
• Recursive Whitening Transformation for Speaker Recognition on Language Mismatched Condition
• The DKU-SMIIP System for NIST 2018 Speaker Recognition Evaluation
• ShaneRun System Description to VoxCeleb Speaker Recognition Challenge 2020
• Audio-visual Speaker Recognition with a Cross-modal Discriminative Network
• HLT-NUS Submission for NIST 2019 Multimedia Speaker Recognition Evaluation
• The DKU System for the Speaker Recognition Task of the 2019 VOiCES from a Distance Challenge
• Speaker Verification By Partial AUC Optimization With Mahalanobis Distance Metric Learning
• Generative Modelling for Unsupervised Score Calibration
• THUEE system description for NIST 2019 SRE CTS Challenge
• Generalization of Spoofing Countermeasures: a Case Study with ASVspoof 2015 and BTAS 2016 Corpora
• Investigation of Synthetic Speech Detection Using Frame- and Segment-Specific Importance Weighting
• STC speaker recognition systems for the NIST SRE 2021