COCOA: Cross Modality Contrastive Learning for Sensor Data

Abstract

Self-Supervised Learning (SSL) is a new paradigm for learning discriminative representations without labelled data and has reached comparable or even state-of-the-art results in comparison to supervised counterparts. Contrastive Learning (CL) is one of the most well-known approaches in SSL that attempts to learn general, informative representations of data. CL methods have been mostly developed for applications in computer vision and natural language processing where only a single sensor modality is used. A majority of pervasive computing applications, however, exploit data from a range of different sensor modalities. While existing CL methods are limited to learning from one or two data sources, we propose COCOA (Cross mOdality COntrastive leArning), a self-supervised model that employs a novel objective function to learn quality representations from multisensor data by computing the cross-correlation between different data modalities and minimizing the similarity between irrelevant instances. We evaluate the effectiveness of COCOA against eight recently introduced state-of-the-art self-supervised models, and two supervised baselines across five public datasets. We show that COCOA achieves superior classification performance to all other approaches. Also, COCOA is far more label-efficient than the other baselines including the fully supervised model using only one-tenth of available labelled data.

arXiv comment27 pages, 10 figures, 6 tables, Accepted with minor revision at IMWUT Vol. 6 No. 3

References in corpus (35)

• Adam: A Method for Stochastic Optimization
• Scikit-learn: Machine Learning in Python
• A Simple Framework for Contrastive Learning of Visual Representations
• Representation Learning with Contrastive Predictive Coding
• Bootstrap your own latent: A new approach to self-supervised Learning
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Unsupervised Representation Learning by Predicting Image Rotations
• DeepSleepNet: a Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG
• Data-Efficient Image Recognition with Contrastive Predictive Coding
• Barlow Twins: Self-Supervised Learning via Redundancy Reduction
• Time Series Data Augmentation for Deep Learning: A Survey
• Self-supervised ECG Representation Learning for Emotion Recognition
• Multi-task Self-Supervised Learning for Human Activity Detection
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech
• Fast Parallel Hypertree Decompositions in Logarithmic Recursion Depth
• Hard Negative Mixing for Contrastive Learning
• Self-Supervised Learning by Cross-Modal Audio-Video Clustering
• Unsupervised Scalable Representation Learning for Multivariate Time Series
• CLEAR: Contrastive Learning for Sentence Representation
• Self-Supervised MultiModal Versatile Networks
• Self-supervised Pretraining of Visual Features in the Wild
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding
• Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligence
• WenLan: Bridging Vision and Language by Large-Scale Multi-Modal Pre-Training
• Subject-Aware Contrastive Learning for Biosignals
• Exploring Contrastive Learning in Human Activity Recognition for Healthcare
• Active Contrastive Learning of Audio-Visual Video Representations
• ESPRESSO: Entropy and ShaPe awaRe timE-Series SegmentatiOn for processing heterogeneous sensor data
• Exploring Self-Supervised Representation Ensembles for COVID-19 Cough Classification
• Weakly Supervised Multi-Task Representation Learning for Human Activity Analysis Using Wearables
• Beyond Just Vision: A Review on Self-Supervised Representation Learning on Multimodal and Temporal Data
• Multimodal Self-Supervised Learning of General Audio Representations
• A Note on Connecting Barlow Twins with Negative-Sample-Free Contrastive Learning

Cited by in corpus (6)

• Diffusion Model-based Contrastive Learning for Human Activity Recognition
• Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Data
• Virtual Fusion with Contrastive Learning for Single Sensor-based Activity Recognition
• Auxiliary Cross-Modal Representation Learning with Triplet Loss Functions for Online Handwriting Recognition
• SudokuSens: Enhancing Deep Learning Robustness for IoT Sensing Applications using a Generative Approach
• MicroCam: Leveraging Smartphone Microscope Camera for Context-Aware Contact Surface Sensing