7 papers
Recent Advances in Multimodal Affective Computing: An NLP Perspective
Guimin Hu, Weimin Lyu, Chang Sun +5
Multimodal affective computing has gained increasing attention due to its broad applications in understanding human behavior and intentions, particularly in text-centric multimodal…
ViDscribe: Multimodal AI for Customizing Audio Description and Question Answering in Online Videos
Maryam Cheema, Sina Elahimanesh, Pooyan Fazli +1
Advances in multimodal large language models enable automatic video narration and question answering (VQA), offering scalable alternatives to labor-intensive, human-authored audio…
HapticLLaMA: A Multimodal Sensory Language Model for Haptic Captioning
Guimin Hu, Daniel Hershcovich, Hasti Seifi
Haptic captioning is the task of generating natural language descriptions from haptic signals, such as vibrations, for use in virtual reality, accessibility, and rehabilitation app…
HapticCap: A Multimodal Dataset and Task for Understanding User Experience of Vibration Haptic Signals
Guimin Hu, Daniel Hershcovich, Hasti Seifi
Haptic signals, from smartphone vibrations to virtual reality touch feedback, can effectively convey information and enhance realism, but designing signals that resonate meaningful…
Partitioner Guided Modal Learning Framework
Guimin Hu, Yi Xin, Lijie Hu +2
Multimodal learning benefits from multiple modal information, and each learned modal representations can be divided into uni-modal that can be learned from uni-modal training and p…
Grounding Emotional Descriptions to Electrovibration Haptic Signals
Guimin Hu, Zirui Zhao, Lukas Heilmann +2
Designing and displaying haptic signals with sensory and emotional attributes can improve the user experience in various applications. Free-form user language provides rich sensory…