Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks

Abstract

Self-attention network (SAN) can benefit significantly from the bi-directional representation learning through unsupervised pretraining paradigms such as BERT and XLNet. In this paper, we present an XLNet-like pretraining scheme "Speech-XLNet" for unsupervised acoustic model pretraining to learn speech representations with SAN. The pretrained SAN is finetuned under the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame orders, the permutation in Speech-XLNet serves as a strong regularizer to encourage the SAN to make inferences by focusing on global structures through its attention weights. In addition, Speech-XLNet also allows the model to explore the bi-directional contexts for effective speech representation learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly improves the SAN/HMM performance in terms of both convergence speed and recognition accuracy compared to the one trained from randomly initialized weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on the TIMIT and WSJ tasks respectively. In particular, the best system achieves a phone error rate (PER) of 13.3% on the TIMIT test set, which to our best knowledge, is the lowest PER obtained from a single system.

arXiv comment\c{opyright} 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders
• vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations
• Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognition
• Self-Attention Networks for Connectionist Temporal Classification in Speech Recognition
• Unsupervised Representation Learning with Future Observation Prediction for Speech Emotion Recognition

Cited by in corpus (5)

• Audio ALBERT: A Lite BERT for Self-supervised Learning of Audio Representation
• A Further Study of Unsupervised Pre-training for Transformer Based Speech Recognition
• Probing Acoustic Representations for Phonetic Properties
• Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
• Input-independent Attention Weights Are Expressive Enough: A Study of Attention in Self-supervised Audio Transformers