Self-supervised ECG Representation Learning for Emotion Recognition

Abstract

We exploit a self-supervised deep multi-task learning framework for electrocardiogram (ECG) -based emotion recognition. The proposed solution consists of two stages of learning a) learning ECG representations and b) learning to classify emotions. ECG representations are learned by a signal transformation recognition network. The network learns high-level abstract representations from unlabeled ECG data. Six different signal transformations are applied to the ECG signals, and transformation recognition is performed as pretext tasks. Training the model on pretext tasks helps the network learn spatiotemporal representations that generalize well across different datasets and different emotion categories. We transfer the weights of the self-supervised network to an emotion recognition network, where the convolutional layers are kept frozen and the dense layers are trained with labelled ECG data. We show that the proposed solution considerably improves the performance compared to a network trained using fully-supervised learning. New state-of-the-art results are set in classification of arousal, valence, affective states, and stress for the four utilized datasets. Extensive experiments are performed, providing interesting insights into the impact of using a multi-task self-supervised structure instead of a single-task model, as well as the optimum level of difficulty required for the pretext self-supervised tasks.

arXiv commentAccepted in IEEE Transactions of Affective Computing

References in corpus (10)

• Adam: A Method for Stochastic Optimization
• Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks
• Multi-task Self-Supervised Learning for Human Activity Detection
• Self-Supervised Spatiotemporal Feature Learning via Video Rotation Prediction
• Self-supervised Learning for ECG-based Emotion Recognition
• Capsule Attention for Multimodal EEG-EOG Representation Learning with Application to Driver Vigilance Estimation
• A Bayesian Deep Learning Framework for End-To-End Prediction of Emotion from Heartbeat
• Multimodal Emotion Recognition Using Deep Canonical Correlation Analysis
• Classification of Cognitive Load and Expertise for Adaptive Simulation using Deep Multitask Learning
• Self-supervised audio representation learning for mobile devices

Cited by in corpus (24)

• Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification
• Self-supervised representation learning from 12-lead ECG data
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• PARSE: Pairwise Alignment of Representations in Semi-Supervised EEG Learning for Emotion Recognition
• A Transformer Architecture for Stress Detection from ECG
• Multimodal Brain-Computer Interface for In-Vehicle Driver Cognitive Load Measurement: Dataset and Baselines
• Detection of Maternal and Fetal Stress from the Electrocardiogram with Self-Supervised Representation Learning
• Label-efficient Time Series Representation Learning: A Review
• Time-Series Representation Learning via Temporal and Contextual Contrasting
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• Towards Domain Generalization for ECG and EEG Classification: Algorithms and Benchmarks
• CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients
• In-Distribution and Out-of-Distribution Self-supervised ECG Representation Learning for Arrhythmia Detection
• On the Generalizability of ECG-based Stress Detection Models
• Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion
• Exploring the Landscape of Ubiquitous In-home Health Monitoring: A Comprehensive Survey
• 3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations
• Automated Contrastive Learning Strategy Search for Time Series
• Semi-supervised Learning for Identifying the Likelihood of Agitation in People with Dementia
• Seismocardiography for Emotion Recognition: A Study on EmoWear with Insights from DEAP
• Which Augmentation Should I Use? An Empirical Investigation of Augmentations for Self-Supervised Phonocardiogram Representation Learning
• Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition
• Evaluating Contrastive Learning on Wearable Timeseries for Downstream Clinical Outcomes