Multi-task Self-Supervised Learning for Human Activity Detection

Abstract

Deep learning methods are successfully used in applications pertaining to ubiquitous computing, health, and well-being. Specifically, the area of human activity recognition (HAR) is primarily transformed by the convolutional and recurrent neural networks, thanks to their ability to learn semantic representations from raw input. However, to extract generalizable features, massive amounts of well-curated data are required, which is a notoriously challenging task; hindered by privacy issues, and annotation costs. Therefore, unsupervised representation learning is of prime importance to leverage the vast amount of unlabeled data produced by smart devices. In this work, we propose a novel self-supervised technique for feature learning from sensory data that does not require access to any form of semantic labels. We learn a multi-task temporal convolutional network to recognize transformations applied on an input signal. By exploiting these transformations, we demonstrate that simple auxiliary tasks of the binary classification result in a strong supervisory signal for extracting useful features for the downstream task. We extensively evaluate the proposed approach on several publicly available datasets for smartphone-based HAR in unsupervised, semi-supervised, and transfer learning settings. Our method achieves performance levels superior to or comparable with fully-supervised networks, and it performs significantly better than autoencoders. Notably, for the semi-supervised case, the self-supervised features substantially boost the detection rate by attaining a kappa score between 0.7-0.8 with only 10 labeled examples per class. We get similar impressive performance even if the features are transferred from a different data source. While this paper focuses on HAR as the application domain, the proposed technique is general and could be applied to a wide variety of problems in other areas.

References in corpus (19)

• Sequence to Sequence Learning with Neural Networks
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• Deep Learning for Sensor-based Activity Recognition: A Survey
• Unsupervised Representation Learning by Predicting Image Rotations
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Data Augmentation of Wearable Sensor Data for Parkinson's Disease Monitoring using Convolutional Neural Networks
• Multi-Scale Convolutional Neural Networks for Time Series Classification
• Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
• Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization
• On the importance of single directions for generalization
• SoundNet: Learning Sound Representations from Unlabeled Video
• SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability
• Realistic Evaluation of Deep Semi-Supervised Learning Algorithms
• Protecting Sensory Data against Sensitive Inferences
• Towards Using Unlabeled Data in a Sparse-coding Framework for Human Activity Recognition
• Measuring the Intrinsic Dimension of Objective Landscapes
• Colorization as a Proxy Task for Visual Understanding
• Learning to Exploit Invariances in Clinical Time-Series Data using Sequence Transformer Networks
• Personalized Driver Stress Detection with Multi-task Neural Networks using Physiological Signals

Cited by in corpus (38)

• Self-supervised ECG Representation Learning for Emotion Recognition
• Self-supervised Contrastive Representation Learning for Semi-supervised Time-Series Classification
• Self-supervised Learning for Human Activity Recognition Using 700,000 Person-days of Wearable Data
• Self-supervised Learning for ECG-based Emotion Recognition
• SelfHAR: Improving Human Activity Recognition through Self-training with Unlabeled Data
• Time Series Change Point Detection with Self-Supervised Contrastive Predictive Coding
• Federated Self-Supervised Learning of Multi-Sensor Representations for Embedded Intelligence
• COCOA: Cross Modality Contrastive Learning for Sensor Data
• Exploring Contrastive Learning in Human Activity Recognition for Healthcare
• Human Activity Recognition using Wearable Sensors: Review, Challenges, Evaluation Benchmark
• ESPRESSO: Entropy and ShaPe awaRe timE-Series SegmentatiOn for processing heterogeneous sensor data
• Self-supervised transfer learning of physiological representations from free-living wearable data
• Label-efficient Time Series Representation Learning: A Review
• Semi-supervised Federated Learning for Activity Recognition
• Time-Series Representation Learning via Temporal and Contextual Contrasting
• Self-supervised Contrastive Learning of Multi-view Facial Expressions
• Diffusion Model-based Contrastive Learning for Human Activity Recognition
• Self-Supervised Time Series Representation Learning by Inter-Intra Relational Reasoning
• Self-supervised Human Activity Recognition by Learning to Predict Cross-Dimensional Motion
• ConvBoost: Boosting ConvNets for Sensor-based Activity Recognition
• A benchmark for computational analysis of animal behavior, using animal-borne tags
• Wearable-based behaviour interpolation for semi-supervised human activity recognition
• DAPPER: Label-Free Performance Estimation after Personalization for Heterogeneous Mobile Sensing
• IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition
• Reducing Label Dependency in Human Activity Recognition with Wearables: From Supervised Learning to Novel Weakly Self-Supervised Approaches
• Learning Generalizable Physiological Representations from Large-scale Wearable Data
• EarDA: Towards Accurate and Data-Efficient Earable Activity Sensing
• PRIMUS: Pretraining IMU Encoders with Multimodal Self-Supervision
• Achieving Single-Sensor Complex Activity Recognition from Multi-Sensor Training Data
• Invariant Feature Learning for Sensor-based Human Activity Recognition
• Saga: Capturing Multi-granularity Semantics from Massive Unlabelled IMU Data for User Perception
• Federated Attention Autoencoders with a Stochastic Aggregation Scheme for Anomaly Detection
• Transformer-based Self-supervised Multimodal Representation Learning for Wearable Emotion Recognition
• Wearable-based Classification of Running Styles with Deep Learning
• Consistency Training of Multi-exit Architectures for Sensor Data
• Self-supervised Deep Learning for Reading Activity Classification
• Experience Paper: Adopting Activity Recognition in On-demand Food Delivery Business
• ViFiT: Reconstructing Vision Trajectories from IMU and Wi-Fi Fine Time Measurements