Weakly-Supervised Dense Action Anticipation

Abstract

Dense anticipation aims to forecast future actions and their durations for long horizons. Existing approaches rely on fully-labelled data, i.e. sequences labelled with all future actions and their durations. We present a (semi-) weakly supervised method using only a small number of fully-labelled sequences and predominantly sequences in which only the (one) upcoming action is labelled. To this end, we propose a framework that generates pseudo-labels for future actions and their durations and adaptively refines them through a refinement module. Given only the upcoming action label as input, these pseudo-labels guide action/duration prediction for the future. We further design an attention mechanism to predict context-aware durations. Experiments on the Breakfast and 50Salads benchmarks verify our method's effectiveness; we are competitive even when compared to fully supervised state-of-the-art models. We will make our code available at: https://github.com/zhanghaotong1/WSLVideoDenseAnticipation.

arXiv commentBMVC 2021

References in corpus (7)

• mixup: Beyond Empirical Risk Minimization
• Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results
• Weakly-Supervised Neural Text Classification
• Forecasting Action through Contact Representations from First Person Video
• Forecasting future action sequences with attention: a new approach to weakly supervised action forecasting
• Anticipation and next action forecasting in video: an end-to-end model with memory
• Adaptive Affinity Loss and Erroneous Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation