8 papers · 1 filter
Living the Novel: A System for Generating Self-Training Timeline-Aware Conversational Agents from Novels
Yifei Huang, Tianyu Yan, Sitong Gong +5
We present the Living Novel, an end-to-end system that transforms any literary work into an immersive, multi-character conversational experience. This system is designed to solve t…
Towards Interactive Intelligence for Digital Humans
Yiyi Cai, Xuangeng Chu, Xiwei Gao +16
We introduce Interactive Intelligence, a novel paradigm of digital human that is capable of personality-aligned expression, adaptive interaction, and self-evolution. To realize thi…
UniLS: End-to-End Audio-Driven Avatars for Unified Listening and Speaking
Xuangeng Chu, Ruicong Liu, Yifei Huang +3
Generating lifelike conversational avatars requires modeling not just isolated speakers, but the dynamic, reciprocal interaction of speaking and listening. However, modeling the li…
Can MLLMs Read the Room? A Multimodal Benchmark for Assessing Deception in Multi-Party Social Interactions
Caixin Kang, Yifei Huang, Liangyang Ouyang +3
Despite their advanced reasoning capabilities, state-of-the-art Multimodal Large Language Models (MLLMs) still lack a core component of human intelligence: the ability to "read the…
SFHand: Learning Embodied Manipulation by Streaming Egocentric 3D Hand Forecasting
Ruicong Liu, Yifei Huang, Liangyang Ouyang +2
Real-time 3D hand forecasting is a critical component for fluid human-computer interaction in applications like AR and assistive robotics. However, existing methods are ill-suited…
AssemblyHands-X: Modeling 3D Hand-Body Coordination for Understanding Bimanual Human Activities
Tatsuro Banno, Takehiko Ohkawa, Ruicong Liu +2
Bimanual human activities inherently involve coordinated movements of both hands and body. However, the impact of this coordination in activity understanding has not been systemati…
Leveraging RGB Images for Pre-Training of Event-Based Hand Pose Estimation
Ruicong Liu, Takehiko Ohkawa, Tze Ho Elden Tse +3
This paper presents RPEP, the first pre-training method for event-based 3D hand pose estimation using labeled RGB images and unpaired, unlabeled event data. Event data offer signif…
Egocentric Action-aware Inertial Localization in Point Clouds with Vision-Language Guidance
Mingfang Zhang, Ryo Yonetani, Yifei Huang +3
This paper presents a novel inertial localization framework named Egocentric Action-aware Inertial Localization (EAIL), which leverages egocentric action cues from head-mounted IMU…