Contrast-reconstruction Representation Learning for Self-supervised Skeleton-based Action Recognition

Abstract

Skeleton-based action recognition is widely used in varied areas, e.g., surveillance and human-machine interaction. Existing models are mainly learned in a supervised manner, thus heavily depending on large-scale labeled data which could be infeasible when labels are prohibitively expensive. In this paper, we propose a novel Contrast-Reconstruction Representation Learning network (CRRL) that simultaneously captures postures and motion dynamics for unsupervised skeleton-based action recognition. It mainly consists of three parts: Sequence Reconstructor, Contrastive Motion Learner, and Information Fuser. The Sequence Reconstructor learns representation from skeleton coordinate sequence via reconstruction, thus the learned representation tends to focus on trivial postural coordinates and be hesitant in motion learning. To enhance the learning of motions, the Contrastive Motion Learner performs contrastive learning between the representations learned from coordinate sequence and additional velocity sequence, respectively. Finally, in the Information Fuser, we explore varied strategies to combine the Sequence Reconstructor and Contrastive Motion Learner, and propose to capture postures and motions simultaneously via a knowledge-distillation based fusion strategy that transfers the motion learning from the Contrastive Motion Learner to the Sequence Reconstructor. Experimental results on several benchmarks, i.e., NTU RGB+D 60, NTU RGB+D 120, CMU mocap, and NW-UCLA, demonstrate the promise of the proposed CRRL method by far outperforming state-of-the-art approaches.

Publised in IEEE TIP. (https://ieeexplore.ieee.org/document/9901454)

References in corpus (9)

• Distilling the Knowledge in a Neural Network
• Bootstrap your own latent: A new approach to self-supervised Learning
• Recurrent Neural Network Regularization
• Improved Baselines with Momentum Contrastive Learning
• NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding
• Self-supervised Learning: Generative or Contrastive
• What Makes for Good Views for Contrastive Learning?
• MS${}^2$L: Multi-Task Self-Supervised Learning for Skeleton Based Action Recognition
• Prototypical Contrast and Reverse Prediction: Unsupervised Skeleton Based Action Recognition