Decoding EEG Brain Activity for Multi-Modal Natural Language Processing
arXiv:2102.08655 · doi:10.3389/fnhum.2021.659410
Abstract
Until recently, human behavioral data from reading has mainly been of interest to researchers to understand human cognition. However, these human language processing signals can also be beneficial in machine learning-based natural language processing tasks. Using EEG brain activity to this purpose is largely unexplored as of yet. In this paper, we present the first large-scale study of systematically analyzing the potential of EEG brain activity data for improving natural language processing tasks, with a special focus on which features of the signal are most beneficial. We present a multi-modal machine learning architecture that learns jointly from textual input as well as from EEG features. We find that filtering the EEG signals into frequency bands is more beneficial than using the broadband signal. Moreover, for a range of word embedding types, EEG data improves binary and ternary sentiment classification and outperforms multiple baselines. For more complex tasks such as relation detection, further research is needed. Finally, EEG data shows to be particularly promising when limited training data is available.
References in corpus (13)
- Deep learning with convolutional neural networks for EEG decoding and visualization
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- InceptionTime: Finding AlexNet for Time Series Classification
- Comparative Study of CNN and RNN for Natural Language Processing
- Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)
- Efficient Large-Scale Multi-Modal Classification
- Advancing NLP with Cognitive Language Processing Signals
- Multiview Deep Learning for Predicting Twitter Users' Location
- Improving Sentiment Analysis with Multi-task Learning of Negation
- Brain2Word: Decoding Brain Activity for Language Generation
- Human brain activity for machine attention
- Data-driven models and computational tools for neurolinguistics: a language technology perspective