283 citations · 682 across the 52 of their papers we have counts for
61 papers · 1 filter
Retrieval-Augmented Online Textual Memory for Episodic Memory Video Question Answering
Raffaele Cali, Giuseppe Lando, Rosario Forte +1
Long egocentric videos pose a central challenge for multimodal language models: the visual stream must be compressed into a memory representation that can be searched efficiently w…
EgoTools: Towards Tool-Centric Reasoning in Real-World Egocentric Videos
Shulin Tian, Junsu Kim, Shuai Liu +17
Real-world embodied tasks, from everyday activities to professional procedures, require agents to act under physical constraints while tracking evolving object and task states. Too…
SLU-2K: A Question-Based Benchmark for Semantic Evaluation of Sign Language Translation
Zeno Testa, Antonino Furnari, Lorenzo Baraldi +1
Sign Language Translation (SLT) is typically evaluated with surface-form metrics such as BLEU and ROUGE, which reward lexical overlap but do not directly measure whether a translat…
EGOSTREAM: A Diagnostic Benchmark for Streaming Episodic Memory in Egocentric Vision
Rosario Forte, Giuseppe Lando, Antonino Furnari
Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools fo…
Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark
Maria Santos-Villafranca, Jesus Bermudez-cameo, Alejandro Perez-Yus +2
To operate in the physical world, embodied agents must perceive their environment in an "always-on" fashion, selectively accessing the most informative sensors to balance energy co…
RECIPE: Procedural Planning via Grounding in Instructional Video
Luigi Seminara, Antonino Furnari, Lorenzo Torresani
Visual planning asks a model to generate the remaining steps of a procedure in natural language given a partial video context and a goal. Progress on this task is bottlenecked by a…
Leveraging Synthetic Data for Enhancing Egocentric Hand-Object Interaction Detection
Rosario Leonardi, Antonino Furnari, Francesco Ragusa +1
In this work, we explore the role of synthetic data in improving the detection of Hand-Object Interactions from egocentric images. Through extensive experimentation and comparative…
ENIGMA-360: An Ego-Exo Dataset for Human Behavior Understanding in Industrial Scenarios
Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto +6
Understanding human behavior from complementary egocentric (ego) and exocentric (exo) points of view enables the development of systems that can support workers in industrial envir…
ViterbiPlanNet: Injecting Procedural Knowledge via Differentiable Viterbi for Planning in Instructional Videos
Luigi Seminara, Davide Moltisanti, Antonino Furnari
Procedural planning aims to predict a sequence of actions that transforms an initial visual state into a desired goal, a fundamental ability for intelligent agents operating in com…
Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation
Lorenzo Mur Labadia, Ruben Martinez-Cantin, Jose J. Guerrero +2
Short Term object-interaction Anticipation consists in detecting the location of the next active objects, the noun and verb categories of the interaction, as well as the time to co…
Exploring Multimodal LMMs for Online Episodic Memory Question Answering on the Edge
Giuseppe Lando, Rosario Forte, Antonino Furnari
We investigate the feasibility of using Multimodal Large Language Models (MLLMs) for real-time online episodic memory question answering. While cloud offloading is common, it raise…
ProSkill: Segment-Level Skill Assessment in Procedural Videos
Michele Mazzamuto, Daniele Di Mauro, Gianpiero Francesca +2
Skill assessment in procedural videos is crucial for the objective evaluation of human performance in settings such as manufacturing and procedural daily tasks. Current research on…
Ego-EXTRA: video-language Egocentric Dataset for EXpert-TRAinee assistance
Francesco Ragusa, Michele Mazzamuto, Rosario Forte +5
We present Ego-EXTRA, a video-language Egocentric Dataset for EXpert-TRAinee assistance. Ego-EXTRA features 50 hours of unscripted egocentric videos of subjects performing procedur…
Learning Egocentric In-Hand Object Segmentation through Weak Supervision from Human Narrations
Nicola Messina, Rosario Leonardi, Luca Ciampi +4
Pixel-level recognition of objects manipulated by the user from egocentric images enables key applications spanning assistive technologies, industrial safety, and activity monitori…
Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video
Alessandro Sebastiano Catinello, Giovanni Maria Farinella, Antonino Furnari
This work tackles the problem of Online detection of Take and Release (OTR) of an object in untrimmed egocentric videos. This task is challenging due to severe label imbalance, wit…
Efficient Calisthenics Skills Classification through Foreground Instance Selection and Depth Estimation
Antonio Finocchiaro, Giovanni Maria Farinella, Antonino Furnari
Calisthenics skill classification is the computer vision task of inferring the skill performed by an athlete from images, enabling automatic performance assessment and personalized…
Calisthenics Skills Temporal Video Segmentation
Antonio Finocchiaro, Giovanni Maria Farinella, Antonino Furnari
Calisthenics is a fast-growing bodyweight discipline that consists of different categories, one of which is focused on skills. Skills in calisthenics encompass both static and dyna…
A Real-Time System for Egocentric Hand-Object Interaction Detection in Industrial Domains
Antonio Finocchiaro, Alessandro Sebastiano Catinello, Michele Mazzamuto +3
Hand-object interaction detection remains an open challenge in real-time applications, where intuitive user experiences depend on fast and accurate detection of interactions with s…
How Far Can Off-the-Shelf Multimodal Large Language Models Go in Online Episodic Memory Question Answering?
Giuseppe Lando, Rosario Forte, Giovanni Maria Farinella +1
We investigate whether off-the-shelf Multimodal Large Language Models (MLLMs) can tackle Online Episodic-Memory Video Question Answering (OEM-VQA) without additional training. Our…
EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs
Ivan Rodin, Tz-Ying Wu, Kyle Min +4
We introduce EASG-Bench, a question-answering benchmark for egocentric videos where the question-answering pairs are created from spatio-temporally grounded dynamic scene graphs ca…
Task Graph Maximum Likelihood Estimation for Procedural Activity Understanding in Egocentric Videos
Luigi Seminara, Giovanni Maria Farinella, Antonino Furnari
We introduce a gradient-based approach for learning task graphs from procedural activities, improving over hand-crafted methods. Our method directly optimizes edge weights via maxi…
Online Episodic Memory Visual Query Localization with Egocentric Streaming Object Memory
Zaira Manigrasso, Matteo Dunnhofer, Antonino Furnari +6
Episodic memory retrieval enables wearable cameras to recall objects or events previously observed in video. However, existing formulations assume an "offline" setting with full vi…
Semantically Guided Action Anticipation
Anxhelo Diko, Antonino Furnari, Luigi Cinque +1
Unsupervised domain adaptation remains a critical challenge in enabling the knowledge transfer of models across unseen domains. Existing methods struggle to balance the need for do…
TI-PREGO: Chain of Thought and In-Context Learning for Online Mistake Detection in PRocedural EGOcentric Videos
Leonardo Plini, Luca Scofano, Edoardo De Matteis +6
Identifying procedural errors online from egocentric videos is a critical yet challenging task across various domains, including manufacturing, healthcare, and skill-based training…