565 citations · 2.8k across the 120 of their papers we have counts for
22 papers · 1 filter
Video Active Perception: Effective Inference-Time Long-Form Video Understanding with Vision-Language Models
Martin Q. Ma, Willis Guo, Aditya Agrawal +4
Large vision-language models (VLMs) have advanced multimodal tasks such as video question answering (QA). However, VLMs face the challenge of selecting frames effectively and effic…
Act2See: Emergent Active Visual Perception for Video Reasoning
Martin Q. Ma, Yuxiao Qu, Aditya Agrawal +4
Vision-Language Models (VLMs) typically rely on static initial frames for video reasoning, restricting their ability to incorporate essential dynamic information as the reasoning p…
ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
Zifu Wan, Ce Zhang, Silong Yong +6
Recent Large Vision-Language Models (LVLMs) have introduced a new paradigm for understanding and reasoning about image input through textual responses. Although they have achieved…
Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset
Vasu Agrawal, Akinniyi Akinyemi, Kathryn Alvero +81
Human communication involves a complex interplay of verbal and nonverbal signals, essential for conveying meaning and achieving interpersonal goals. To develop socially intelligent…
OpenFace 3.0: A Lightweight Multitask System for Comprehensive Facial Behavior Analysis
Jiewen Hu, Leena Mathur, Paul Pu Liang +1
In recent years, there has been increasing interest in automatic facial behavior analysis systems from computing communities such as vision, multimodal interaction, robotics, and a…
AV-Flow: Transforming Text to Audio-Visual Human-like Interactions
Aggelina Chatziagapi, Louis-Philippe Morency, Hongyu Gong +3
We introduce AV-Flow, an audio-visual generative model that animates photo-realistic 4D talking avatars given only text input. In contrast to prior work that assumes an existing sp…
Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Models
Ce Zhang, Zifu Wan, Zhehan Kan +7
While recent Large Vision-Language Models (LVLMs) have shown remarkable performance in multi-modal tasks, they are prone to generating hallucinatory text responses that do not alig…
Paraphrasing Is All You Need for Novel Object Captioning
Cheng-Fu Yang, Yao-Hung Hubert Tsai, Wan-Cyuan Fan +3
Novel object captioning (NOC) aims to describe images containing objects without observing their ground truth captions during training. Due to the absence of caption annotation, ca…
Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approach
Chaitanya Ahuja, Dong Won Lee, Yukiko I. Nakano +1
How can we teach robots or virtual assistants to gesture naturally? Can we go further and adapt the gesturing style to follow a specific speaker? Gestures that are naturally timed…
Diverse and Admissible Trajectory Forecasting through Multimodal Context Understanding
Seong Hyeon Park, Gyubok Lee, Manoj Bhat +6
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable…
Context-Dependent Models for Predicting and Characterizing Facial Expressiveness
Victoria Lin, Jeffrey M. Girard, Louis-Philippe Morency
In recent years, extensive research has emerged in affective computing on topics like automatic emotion recognition and determining the signals that characterize individual emotion…
To React or not to React: End-to-End Visual Pose Forecasting for Personalized Avatar during Dyadic Conversations
Chaitanya Ahuja, Shugao Ma, Louis-Philippe Morency +1
Non verbal behaviours such as gestures, facial expressions, body posture, and para-linguistic cues have been shown to complement or clarify verbal messages. Hence to improve telepr…
Language2Pose: Natural Language Grounded Pose Forecasting
Chaitanya Ahuja, Louis-Philippe Morency
Generating animations from natural language sentences finds its applications in a a number of domains such as movie script visualization, virtual human animation and, robot motion…
Video Relationship Reasoning using Gated Spatio-Temporal Energy Graph
Yao-Hung Hubert Tsai, Santosh Divvala, Louis-Philippe Morency +2
Visual relationship reasoning is a crucial yet challenging task for understanding rich interactions across visual concepts. For example, a relationship 'man, open, door' involves a…
Multimodal Polynomial Fusion for Detecting Driver Distraction
Yulun Du, Chirag Raman, Alan W Black +2
Distracted driving is deadly, claiming 3,477 lives in the U.S. in 2015 alone. Although there has been a considerable amount of research on modeling the distracted behavior of drive…
Speaker-Follower Models for Vision-and-Language Navigation
Daniel Fried, Ronghang Hu, Volkan Cirik +7
Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few h…
Using Syntax to Ground Referring Expressions in Natural Images
Volkan Cirik, Taylor Berg-Kirkpatrick, Louis-Philippe Morency
We introduce GroundNet, a neural network for referring expression recognition -- the task of localizing (or grounding) in an image the object referred to by a natural language expr…
Hand2Face: Automatic Synthesis and Recognition of Hand Over Face Occlusions
Behnaz Nojavanasghari, Charles. E. Hughes, Tadas Baltrusaitis +1
A person's face discloses important information about their affective state. Although there has been extensive research on recognition of facial expressions, the performance of exi…
GazeDirector: Fully Articulated Eye Gaze Redirection in Video
Erroll Wood, Tadas Baltrusaitis, Louis-Philippe Morency +2
We present GazeDirector, a new approach for eye gaze redirection that uses model-fitting. Our method first tracks the eyes by fitting a multi-part eye region model to video frames…
Temporal Attention-Gated Model for Robust Sequence Classification
Wenjie Pei, Tadas Baltrušaitis, David M. J. Tax +1
Typical techniques for sequence classification are designed for well-segmented sequences which have been edited to remove noisy or irrelevant parts. Therefore, such methods cannot…
Convolutional Experts Constrained Local Model for Facial Landmark Detection
Amir Zadeh, Tadas Baltrušaitis, Louis-Philippe Morency
Constrained Local Models (CLMs) are a well-established family of methods for facial landmark detection. However, they have recently fallen out of favor to cascaded regression-based…
Relative Facial Action Unit Detection
Mahmoud Khademi, Louis-Philippe Morency
This paper presents a subject-independent facial action unit (AU) detection method by introducing the concept of relative AU detection, for scenarios where the neutral face is not…