Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding

Abstract

Online action understanding, encompassing action detection and anticipation, is central to numerous practical applications. However, untrimmed videos contain substantial redundant and noisy historical information, and the complementary interactions among historical, current, and future-oriented representations remain underexplored. To address these challenges, we propose the State-Specific Model (SSM), a unified framework for online action detection and anticipation. Its Representative State-Based Memory Compression module selects representative anchor frames from the observed memory and aggregates anchor-conditioned context into a compact set of representative memory states, thereby reducing memory redundancy. The subsequent Action Pattern Learning module learns multi-dimensional relations among these states and performs graph reasoning over the resulting directed representative-state graph to model action dynamics and derive a future-oriented cue. The Cross-Temporal Interaction module then enables this cue to interact with historical and current cues, refining the representations used for simultaneous online action detection and anticipation. Extensive experiments on EPIC-Kitchens-100, THUMOS'14, TVSeries, and the introduced Parkinson's Disease Mouse Behaviour dataset demonstrate competitive performance and generalizability across diverse online action understanding settings. These results underscore the value of graph-based action-dynamics modelling and cross-temporal interaction for online action understanding.