FlatClip: Reusing Image Foundation Models for fMRI Representation Learning via Cortical Flatmaps

Abstract

Can the spatial organization of cortical activity help transfer frozen image representations to fMRI prediction? We introduce FlatClip, which renders cortical activity as anatomy-linked flatmaps and extracts features with a frozen SigLIP2 encoder. Only a lightweight downstream probe is trained, without fMRI-specific encoder pretraining or fine-tuning. Across four resting-state benchmarks, FlatClip outperforms the evaluated ROI-level baselines on HCP and ADNI, but is weaker on PPMI and below the strongest voxel-level models overall. On NSD visual-fMRI decoding, restricting inputs to visual or NSD-provided task-active cortex improves recognition over whole-cortex inputs. Spatial perturbations reduce performance with both retrained and fixed readouts, and anatomy-linked arrangements outperform cortical-pixel permutations across three colormaps. Together, these results demonstrate a practical route for reusing image-pretrained features in fMRI and highlight cortical arrangement and input-region selection as important design choices. Code is available at https://github.com/OneMore1/FlatClip.

NeurIPS 2026