An Open source Implementation of ITU-T Recommendation P.808 with Validation

Abstract

The ITU-T Recommendation P.808 provides a crowdsourcing approach for conducting a subjective assessment of speech quality using the Absolute Category Rating (ACR) method. We provide an open-source implementation of the ITU-T Rec. P.808 that runs on the Amazon Mechanical Turk platform. We extended our implementation to include Degradation Category Ratings (DCR) and Comparison Category Ratings (CCR) test methods. We also significantly speed up the test process by integrating the participant qualification step into the main rating task compared to a two-stage qualification and rating solution. We provide program scripts for creating and executing the subjective test, and data cleansing and analyzing the answers to avoid operational errors. To validate the implementation, we compare the Mean Opinion Scores (MOS) collected through our implementation with MOS values from a standard laboratory experiment conducted based on the ITU-T Rec. P.800. We also evaluate the reproducibility of the result of the subjective speech quality assessment through crowdsourcing using our implementation. Finally, we quantify the impact of parts of the system designed to improve the reliability: environmental tests, gold and trapping questions, rating patterns, and a headset usage test.

References in corpus (3)

• The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Speech Quality and Testing Framework
• Application of Just-Noticeable Difference in Quality as Environment Suitability Test for Crowdsourcing Speech Quality Assessment Task
• Impact of the Number of Votes on the Reliability and Validity of Subjective Speech Quality Assessment in the Crowdsourcing Approach

Cited by in corpus (16)

• NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets
• Speech Enhancement and Dereverberation with Diffusion-based Generative Models
• F-T-LSTM based Complex Network for Joint Acoustic Echo Cancellation and Speech Enhancement
• A Comparative Study of Self-supervised Speech Representation Based Voice Conversion
• Comparison of remote experiments using crowdsourcing and laboratory experiments on speech intelligibility
• Crowdsourced Multilingual Speech Intelligibility Testing
• Pre-trained Speech Representations as Feature Extractors for Speech Quality Assessment in Online Conferencing Applications
• Pretraining Techniques for Sequence-to-Sequence Voice Conversion
• Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement
• Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations
• Weighted Recursive Least Square Filter and Neural Network based Residual Echo Suppression for the AEC-Challenge
• Acoustic Echo Cancellation using Residual U-Nets
• On Prosody Modeling for ASR+TTS based Voice Conversion
• How do Voices from Past Speech Synthesis Challenges Compare Today?
• UBGAN: Enhancing Coded Speech with Blind and Guided Bandwidth Extension
• Towards Reliable Objective Evaluation Metrics for Generative Singing Voice Separation Models