UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Models

Abstract

Non-autoregressive automatic speech recognition (NASR) models have gained attention due to their parallelism and fast inference. The encoder-based NASR, e.g. connectionist temporal classification (CTC), can be initialized from the speech foundation models (SFM) but does not account for any dependencies among intermediate tokens. The encoder-decoder-based NASR, like CTC alignment-based single-step non-autoregressive transformer (CASS-NAT), can mitigate the dependency problem but is not able to efficiently integrate SFM. Inspired by the success of recent work of speech-text joint pre-training with a shared transformer encoder, we propose a new encoder-based NASR, UniEnc-CASSNAT, to combine the advantages of CTC and CASS-NAT. UniEnc-CASSNAT consists of only an encoder as the major module, which can be the SFM. The encoder plays the role of both the CASS-NAT encoder and decoder by two forward passes. The first pass of the encoder accepts the speech signal as input, while the concatenation of the speech signal and the token-level acoustic embedding is used as the input for the second pass. Examined on the Librispeech 100h, MyST, and Aishell1 datasets, the proposed UniEnc-CASSNAT achieves state-of-the-art NASR results and is better or comparable to CASS-NAT with only an encoder and hence, fewer model parameters. Our codes are publicly available.

arXiv commentPublished in IEEE Signal Processing Letters

References in corpus (7)

• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing
• Self-Supervised Speech Representation Learning: A Review
• BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition
• Listen and Fill in the Missing Letters: Non-Autoregressive Transformer for Speech Recognition
• A CTC Alignment-based Non-autoregressive Transformer for End-to-end Automatic Speech Recognition
• Towards Better Domain Adaptation for Self-supervised Models: A Case Study of Child ASR
• Pushing the Limits of Non-Autoregressive Speech Recognition