A Hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation

Abstract

Action recognition has become a rapidly developing research field within the last decade. But with the increasing demand for large scale data, the need of hand annotated data for the training becomes more and more impractical. One way to avoid frame-based human annotation is the use of action order information to learn the respective action classes. In this context, we propose a hierarchical approach to address the problem of weakly supervised learning of human actions from ordered action labels by structuring recognition in a coarse-to-fine manner. Given a set of videos and an ordered list of the occurring actions, the task is to infer start and end frames of the related action classes within the video and to train the respective action classifiers without any need for hand labeled frame boundaries. We address this problem by combining a framewise RNN model with a coarse probabilistic inference. This combination allows for the temporal alignment of long sequences and thus, for an iterative training of both elements. While this system alone already generates good results, we show that the performance can be further improved by approximating the number of subactions to the characteristics of the different action classes as well as by the introduction of a regularizing length prior. The proposed system is evaluated on two benchmark datasets, the Breakfast and the Hollywood extended dataset, showing a competitive performance on various weak learning tasks such as temporal action segmentation and action alignment.

arXiv comment15 pages, preprint for IEEE TPAMI https://ieeexplore.ieee.org/document/8585084 (open access). arXiv admin note: substantial text overlap with arXiv:1703.08132

References in corpus (8)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• Two-Stream Convolutional Networks for Action Recognition in Videos
• UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild
• The Kinetics Human Action Video Dataset
• Temporal Localization of Fine-Grained Actions in Videos by Domain Transfer from Web Images
• Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment
• Inference in Hidden Markov Models with Explicit State Duration Distributions
• State Duration and Interval Modeling in Hidden Semi-Markov Model for Sequential Data Analysis

Cited by in corpus (8)

• Coarse to Fine Multi-Resolution Temporal Convolutional Network
• Facial Tic Detection in Untrimmed Videos of Tourette Syndrome Patients
• Motion2Vec: Semi-Supervised Representation Learning from Surgical Videos
• Localizing the Common Action Among a Few Videos
• Temporal Action Segmentation from Timestamp Supervision
• Discovering Multi-Label Actor-Action Association in a Weakly Supervised Setting
• Background-Click Supervision for Temporal Action Localization
• Global2Local: Efficient Structure Search for Video Action Segmentation