Multimodal Data Fusion based on the Global Workspace Theory
arXiv:2001.09485
Abstract
We propose a novel neural network architecture, named the Global Workspace Network (GWN), which addresses the challenge of dynamic and unspecified uncertainties in multimodal data fusion. Our GWN is a model of attention across modalities and evolving through time, and is inspired by the well-established Global Workspace Theory from the field of cognitive science. The GWN achieved average F1 score of 0.92 for discrimination between pain patients and healthy participants and average F1 score = 0.75 for further classification of three pain levels for a patient, both based on the multimodal EmoPain dataset captured from people with chronic pain and healthy people performing different types of exercise movements in unconstrained settings. In these tasks, the GWN significantly outperforms the typical fusion approach of merging by concatenation. We further provide extensive analysis of the behaviour of the GWN and its ability to address uncertainties (hidden noise) in multimodal data.
arXiv comment12 pages, 5 figures
References in corpus (14)
- Adam: A Method for Stochastic Optimization
- Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting
- Deep Residual Learning for Image Recognition
- A Critical Review of Recurrent Neural Networks for Sequence Learning
- A Deep Reinforced Model for Abstractive Summarization
- Hierarchical Question-Image Co-Attention for Visual Question Answering
- Scalable Tensor Factorizations for Incomplete Data
- Memory Fusion Network for Multi-view Sequential Learning
- An Introductory Survey on Attention Mechanisms in NLP Problems
- Multimodal Attention for Neural Machine Translation
- Heterogeneous Memory Enhanced Multimodal Attention Model for Video Question Answering
- Attention-Based Multimodal Fusion for Video Description
- Effects of padding on LSTMs and CNNs
- Multi-level Multimodal Common Semantic Space for Image-Phrase Grounding