Multimodal neural networks better explain multivoxel patterns in the hippocampus

Abstract

The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.

arXiv commentOral at SVRHM Workshop (NeurIPS 2021)

References in corpus (5)

• Intriguing properties of neural networks
• Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Brain's Ventral Visual Pathway
• HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips
• Self-Supervised MultiModal Versatile Networks
• Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

Cited by in corpus (1)

• Improving deep learning with prior knowledge and cognitive models: A survey on enhancing explainability, adversarial robustness and zero-shot learning