IMUTube: Automatic Extraction of Virtual on-body Accelerometry from Video for Human Activity Recognition
arXiv:2006.05675
Abstract
The lack of large-scale, labeled data sets impedes progress in developing robust and generalized predictive models for on-body sensor-based human activity recognition (HAR). Labeled data in human activity recognition is scarce and hard to come by, as sensor data collection is expensive, and the annotation is time-consuming and error-prone. To address this problem, we introduce IMUTube, an automated processing pipeline that integrates existing computer vision and signal processing techniques to convert videos of human activity into virtual streams of IMU data. These virtual IMU streams represent accelerometry at a wide variety of locations on the human body. We show how the virtually-generated IMU data improves the performance of a variety of models on known HAR datasets. Our initial results are very promising, but the greater promise of this work lies in a collective approach by the computer vision, signal processing, and activity recognition communities to extend this work in ways that we outline. This should lead to on-body, sensor-based HAR becoming yet another success story in large-dataset breakthroughs in recognition.
References in corpus (15)
- Adam: A Method for Stochastic Optimization
- GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
- UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
- Simple Online and Realtime Tracking
- How transferable are features in deep neural networks?
- NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
- Ensembles of Deep LSTM Learners for Activity Recognition using Wearables
- Deep, Convolutional, and Recurrent Models for Human Activity Recognition using Wearables
- A Short Note on the Kinetics-700-2020 Human Action Dataset
- Multi-task Self-Supervised Learning for Human Activity Detection
- Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding
- T-CGAN: Conditional Generative Adversarial Network for Data Augmentation in Noisy Time Series with Irregular Sampling
- Data augmentation using synthetic data for time series classification with deep residual networks
- Distribution Matching in Variational Inference
- A Deep Learning Method for Complex Human Activity Recognition Using Virtual Wearable Sensors