BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognition

Abstract

We introduce BERTphone, a Transformer encoder trained on large speech corpora that outputs phonetically-aware contextual representation vectors that can be used for both speaker and language recognition. This is accomplished by training on two objectives: the first, inspired by adapting BERT to the continuous domain, involves masking spans of input frames and reconstructing the whole sequence for acoustic representation learning; the second, inspired by the success of bottleneck features from ASR, is a sequence-level CTC loss applied to phoneme labels for phonetic representation learning. We pretrain two BERTphone models (one on Fisher and one on TED-LIUM) and use them as feature extractors into x-vector-style DNNs for both tasks. We attain a state-of-the-art $C_{\text{avg}}$ of 6.16 on the challenging LRE07 3sec closed-set language recognition task. On Fisher and VoxCeleb speaker recognition tasks, we see an 18% relative reduction in speaker EER when training on BERTphone vectors instead of MFCCs. In general, BERTphone outperforms previous phonetic pretraining approaches on the same data. We release our code and models at https://github.com/awslabs/speech-representations.

Odyssey 2020 camera-ready (presented Nov. 2020)

References in corpus (7)

• SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition
• A Structured Self-attentive Sentence Embedding
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition

Cited by in corpus (9)

• Self-Supervised Speech Representation Learning: A Review
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• UniSpeech: Unified Speech Representation Learning with Labeled and Unlabeled Data
• Speaker Recognition Based on Deep Learning: An Overview
• UniSpeech at scale: An Empirical Study of Pre-training Method on Large-Scale Speech Recognition Dataset
• Probing Acoustic Representations for Phonetic Properties
• Guided contrastive self-supervised pre-training for automatic speech recognition
• Using Pause Information for More Accurate Entity Recognition
• Multi-Modal Transformers Utterance-Level Code-Switching Detection