MOABB: Trustworthy algorithm benchmarking for BCIs

Abstract

BCI algorithm development has long been hampered by two major issues: small sample sets and a lack of reproducibility. We offer a solution to both of these problems via a software suite that streamlines both the issues of finding and preprocessing data in a reliable manner, as well as that of using a consistent interface for machine learning methods. By building on recent advances in software for signal analysis implemented in the MNE toolkit, and the unified framework for machine learning offered by the scikit-learn project, we offer a system that can improve BCI algorithm development. This system is fully open-source under the BSD licence and available at https://github.com/NeuroTechX/moabb. To validate our efforts, we analyze a set of state-of-the-art decoding algorithms across 12 open access datasets, with over 250 subjects. Our analysis confirms that different datasets can result in very different results for identical processing pipelines, highlighting the need for trustworthy algorithm benchmarking in the field of BCIs, and further that many previously validated methods do not hold up when applied across different datasets, which has wide-reaching implications for practical BCIs.

References in corpus (2)

• Shrinkage Algorithms for MMSE Covariance Estimation
• Transfer Learning in Brain-Computer Interfaces

Cited by in corpus (15)

• Decoding speech perception from non-invasive brain recordings
• CSP-Net: Common Spatial Pattern Empowered Neural Networks for EEG-Based Motor Imagery Classification
• Channel Reflection: Knowledge-Driven Data Augmentation for EEG-Based Brain-Computer Interfaces
• T-TIME: Test-Time Information Maximization Ensemble for Plug-and-Play BCIs
• Federated Motor Imagery Classification for Privacy-Preserving Brain-Computer Interfaces
• BrainVis: Exploring the Bridge between Brain and Visual Signals via Image Reconstruction
• Minimizing subject-dependent calibration for BCI with Riemannian transfer learning
• A Survey of Spatio-Temporal EEG data Analysis: from Models to Applications
• Data augmentation for learning predictive models on EEG: a systematic comparison
• Playing With Neuroscience: Past, Present and Future of Neuroimaging and Games
• The Riemannian Means Field Classifier for EEG-Based BCI Data
• Exploring Deep Learning Models for EEG Neural Decoding
• MIRepNet: A Pipeline and Foundation Model for EEG-Based Motor Imagery Classification
• End-to-end P300 BCI using Bayesian accumulation of Riemannian probabilities
• Spatial Distillation based Distribution Alignment (SDDA) for Cross-Headset EEG Classification