SpeechFormer++: A Hierarchical Efficient Framework for Paralinguistic Speech Processing

Abstract

Paralinguistic speech processing is important in addressing many issues, such as sentiment and neurocognitive disorder analyses. Recently, Transformer has achieved remarkable success in the natural language processing field and has demonstrated its adaptation to speech. However, previous works on Transformer in the speech field have not incorporated the properties of speech, leaving the full potential of Transformer unexplored. In this paper, we consider the characteristics of speech and propose a general structure-based framework, called SpeechFormer++, for paralinguistic speech processing. More concretely, following the component relationship in the speech signal, we design a unit encoder to model the intra- and inter-unit information (i.e., frames, phones, and words) efficiently. According to the hierarchical relationship, we utilize merging blocks to generate features at different granularities, which is consistent with the structural pattern in the speech signal. Moreover, a word encoder is introduced to integrate word-grained features into each unit encoder, which effectively balances fine-grained and coarse-grained information. SpeechFormer++ is evaluated on the speech emotion recognition (IEMOCAP & MELD), depression classification (DAIC-WOZ) and Alzheimer's disease detection (Pitt) tasks. The results show that SpeechFormer++ outperforms the standard Transformer while greatly reducing the computational cost. Furthermore, it delivers superior results compared to the state-of-the-art approaches.

14 pages, 7 figures, 14 tables, TASLP 2023 paper

References in corpus (11)

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• DeepViT: Towards Deeper Vision Transformer
• DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsification
• Transformer-based Acoustic Modeling for Hybrid Speech Recognition
• ResT: An Efficient Transformer for Visual Recognition
• Self-Attention Transducers for End-to-End Speech Recognition
• Key-Sparse Transformer for Multimodal Speech Emotion Recognition
• Detecting Dementia from Speech and Transcripts using Transformers
• BOAT: Bilateral Local Attention Vision Transformer
• A Topic-Attentive Transformer-based Model For Multimodal Depression Detection
• Learning Fine-Grained Cross Modality Excitement for Speech Emotion Recognition