Gestura: A LVLM-Powered System Bridging Motion and Semantics for Real-Time Free-Form Gesture Understanding

Abstract

Free-form gesture understanding is highly appealing for human-computer interaction, as it liberates users from the constraints of predefined gesture categories. However, the sole existing solution GestureGPT suffers from limited recognition accuracy and slow response times. In this paper, we propose Gestura, an end-to-end system for free-form gesture understanding. Gestura harnesses a pre-trained Large Vision-Language Model (LVLM) to align the highly dynamic and diverse patterns of free-form gestures with high-level semantic concepts. To better capture subtle hand movements across different styles, we introduce a Landmark Processing Module that compensate for LVLMs' lack of fine-grained domain knowledge by embedding anatomical hand priors. Further, a Chain-of-Thought (CoT) reasoning strategy enables step-by-step semantic inference, transforming shallow knowledge into deep semantic understanding and significantly enhancing the model's ability to interpret ambiguous or unconventional gestures. Together, these components allow Gestura to achieve robust and adaptable free-form gesture comprehension. Additionally, we have developed the first open-source dataset for free-form gesture intention reasoning and understanding with over 300,000 annotated QA pairs.

arXiv commentIMWUT2025

References in corpus (7)

• Sasha: Creative Goal-Oriented Reasoning in Smart Homes with Large Language Models
• ChaLearn Looking at People: IsoGD and ConGD Large-scale RGB-D Gesture Recognition
• MAPLE: Mobile App Prediction Leveraging Large Language Model Embeddings
• MemX: An Attention-Aware Smart Eyewear System for Personalized Moment Auto-capture
• Optimizing Gesture Recognition for Seamless UI Interaction Using Convolutional Neural Networks
• Exploring Uni-manual Around Ear Off-Device Gestures for Earables
• Towards Open-World Gesture Recognition