sDREAMER: Self-distilled Mixture-of-Modality-Experts Transformer for Automatic Sleep Staging

Abstract

Automatic sleep staging based on electroencephalography (EEG) and electromyography (EMG) signals is an important aspect of sleep-related research. Current sleep staging methods suffer from two major drawbacks. First, there are limited information interactions between modalities in the existing methods. Second, current methods do not develop unified models that can handle different sources of input. To address these issues, we propose a novel sleep stage scoring model sDREAMER, which emphasizes cross-modality interaction and per-channel performance. Specifically, we develop a mixture-of-modality-expert (MoME) model with three pathways for EEG, EMG, and mixed signals with partially shared weights. We further propose a self-distillation training scheme for further information interaction across modalities. Our model is trained with multi-channel inputs and can make classifications on either single-channel or multi-channel inputs. Experiments demonstrate that our model outperforms the existing transformer-based sleep scoring methods for multi-channel inference. For single-channel inference, our model also outperforms the transformer-based models trained with single-channel signals.

References in corpus (18)

• XGBoost: A Scalable Tree Boosting System
• Decoupled Weight Decay Regularization
• DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• SeqSleepNet: End-to-End Hierarchical Recurrent Neural Network for Sequence-to-Sequence Automatic Sleep Staging
• A Time Series is Worth 64 Words: Long-term Forecasting with Transformers
• Joint Classification and Prediction CNN Framework for Automatic Sleep Stage Classification
• Neural network an1alysis of sleep stages enables efficient diagnosis of narcolepsy
• SleepTransformer: Automatic Sleep Staging with Interpretability and Uncertainty Quantification
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation
• VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts
• XSleepNet: Multi-View Sequential Model for Automatic Sleep Staging
• Neural Controlled Differential Equations for Irregular Time Series
• Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learning
• SLEEPNET: Automated Sleep Staging System via Deep Learning
• Self-Distillation Amplifies Regularization in Hilbert Space
• Self-Distillation as Instance-Specific Label Smoothing
• Automated Classification of Sleep Stages and EEG Artifacts in Mice with Deep Learning