Decoding speech perception from non-invasive brain recordings

Abstract

Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in that regard: deep learning algorithms trained on intracranial recordings now start to decode elementary linguistic features (e.g. letters, words, spectrograms). However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here, we introduce a model trained with contrastive-learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto- or electro-encephalography (M/EEG), while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of MEG signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and more than 80% in the very best participants - a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model to a variety of baselines highlights the importance of (i) a contrastive objective, (ii) pretrained representations of speech and (iii) a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder's predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk for brain surgery.

updated version following publication in Nature Machine Intelligence (2023)

References in corpus (7)

• Learning Transferable Visual Models From Natural Language Supervision
• Toward a realistic model of speech processing in the brain with self-supervised learning
• Successes and critical failures of neural networks in capturing human-like speech recognition
• Self-supervised models of audio effectively explain human cortical responses to speech
• Brain2Word: Decoding Brain Activity for Language Generation
• Natural scene reconstruction from fMRI signals using generative latent diffusion
• MEG-MASC: a high-quality magneto-encephalography dataset for evaluating natural speech processing

Cited by in corpus (9)

• ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG
• Group-level Brain Decoding with Deep Learning
• Deep Neural Networks and Brain Alignment: Brain Encoding and Decoding (Survey)
• Wireless Silent Speech Interface Using Multi-Channel Textile EMG Sensors Integrated into Headphones
• Brain-inspired Computing Based on Deep Learning for Human-computer Interaction: A Review
• Deep Representation Learning for Open Vocabulary Electroencephalography-to-Text Decoding
• EEG-MACS: Manifold Attention and Confidence Stratification for EEG-based Cross-Center Brain Disease Diagnosis under Unreliable Annotations
• Exploring Deep Learning Models for EEG Neural Decoding
• R&B -- Rhythm and Brain: Cross-subject Decoding of Music from Human Brain Activity