Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

Abstract

Alzheimer's disease (AD) constitutes a complex neurocognitive disease and is the main cause of dementia. Although many studies have been proposed targeting at diagnosing dementia through spontaneous speech, there are still limitations. Existing state-of-the-art approaches, which propose multimodal methods, train separately language and acoustic models, employ majority-vote approaches, and concatenate the representations of the different modalities either at the input level, i.e., early fusion, or during training. Also, some of them employ self-attention layers, which calculate the dependencies between representations without considering the contextual information. In addition, no prior work has taken into consideration the model calibration. To address these limitations, we propose some new methods for detecting AD patients, which capture the intra- and cross-modal interactions. First, we convert the audio files into log-Mel spectrograms, their delta, and delta-delta and create in this way an image per audio file consisting of three channels. Next, we pass each transcript and image through BERT and DeiT models respectively. After that, context-based self-attention layers, self-attention layers with a gate model, and optimal transport domain adaptation methods are employed for capturing the intra- and inter-modal interactions. Finally, we exploit two methods for fusing the self and cross-attention features. For taking into account the model calibration, we apply label smoothing. We use both performance and calibration metrics. Experiments conducted on the ADReSS and ADReSSo Challenge datasets indicate the efficacy of our introduced approaches over existing research initiatives with our best performing model reaching Accuracy and F1-score up to 91.25% and 91.06% respectively.

arXiv commentKnowledge-Based Systems

References in corpus (6)

• Robust Speech Recognition via Large-Scale Weak Supervision
• Explainable Identification of Dementia from Transcripts using Transformer Networks
• Detecting Dementia from Speech and Transcripts using Transformers
• Multimodal Unified Attention Networks for Vision-and-Language Interactions
• deep-significance - Easy and Meaningful Statistical Significance Testing in the Age of Neural Networks
• Multi-Modal Detection of Alzheimer's Disease from Speech and Text

Cited by in corpus (3)

• A Graph Neural Architecture Search Approach for Identifying Bots in Social Media
• Leveraging Large Language Models through Natural Language Processing to provide interpretable Machine Learning predictions of mental deterioration in real time
• Speech-Based Cognitive Screening: A Systematic Evaluation of LLM Adaptation Strategies