6 papers
Through a PRISM, Clearly: Step-Sequence Reasoning for Mistake Recognition in Procedural Videos
Olga Loginova, Oleksandr Bezrukov, Frank Keller
Procedural mistake recognition requires identifying deviations from valid multi-step activities. We study post-completion recognition from videos with benchmark-provided step bound…
How to Correctly Make Mistakes: A Framework for Constructing and Benchmarking Mistake Aware Egocentric Procedural Videos
Olga Loginova, Frank Keller
Reliable procedural monitoring in video requires exposure to naturally occurring human errors and the recoveries that follow. In egocentric recordings, mistakes are often partially…
Predicting Implicit Arguments in Procedural Video Instructions
Anil Batra, Laura Sevilla-Lara, Marcus Rohrbach +1
Procedural texts help AI enhance reasoning about context and action sequences. Transforming these into Semantic Role Labeling (SRL) improves understanding of individual steps by id…
Coarse or Fine? Recognising Action End States without Labels
Davide Moltisanti, Hakan Bilen, Laura Sevilla-Lara +1
We focus on the problem of recognising the end state of an action in an image, which is critical for understanding what action is performed and in which manner. We study this focus…
Efficient Pre-training for Localized Instruction Generation of Videos
Anil Batra, Davide Moltisanti, Laura Sevilla-Lara +2
Procedural videos, exemplified by recipe demonstrations, are instrumental in conveying step-by-step instructions. However, understanding such videos is challenging as it involves t…
Semi-supervised multimodal coreference resolution in image narrations
Arushi Goel, Basura Fernando, Frank Keller +1
In this paper, we study multimodal coreference resolution, specifically where a longer descriptive text, i.e., a narration is paired with an image. This poses significant challenge…