Mixing Up Contrastive Learning: Self-Supervised Representation Learning for Time Series

Abstract

The lack of labeled data is a key challenge for learning useful representation from time series data. However, an unsupervised representation framework that is capable of producing high quality representations could be of great value. It is key to enabling transfer learning, which is especially beneficial for medical applications, where there is an abundance of data but labeling is costly and time consuming. We propose an unsupervised contrastive learning framework that is motivated from the perspective of label smoothing. The proposed approach uses a novel contrastive loss that naturally exploits a data augmentation scheme in which new samples are generated by mixing two data samples with a mixing component. The task in the proposed framework is to predict the mixing component, which is utilized as soft targets in the loss function. Experiments demonstrate the framework's superior performance compared to other representation learning approaches on both univariate and multivariate time series and illustrate its benefits for transfer learning for clinical time series.

arXiv commentPublished in Journal of Pattern Recognition Letters: https://www.sciencedirect.com/science/article/pii/S0167865522000502 Code available at: https://github.com/Wickstrom/MixupContrastiveLearning

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• A Simple Framework for Contrastive Learning of Visual Representations
• mixup: Beyond Empirical Risk Minimization
• Bootstrap your own latent: A new approach to self-supervised Learning
• When Does Label Smoothing Help?
• Deep Clustering for Unsupervised Learning of Visual Features
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• Adaptive Exploration for Unsupervised Person Re-Identification
• Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learning

Cited by in corpus (5)

• Label-efficient Time Series Representation Learning: A Review
• Self-supervised learning-based general laboratory progress pretrained model for cardiovascular event detection
• A Co-training Approach for Noisy Time Series Learning
• Augmentation-aware Self-supervised Learning with Conditioned Projector
• "It Looks All the Same to Me": Cross-index Training for Long-term Financial Series Prediction