14 citations · 23 across the 23 of their papers we have counts for
28 papers
EgoPC: Internalizing Plan-Perceive-Compose for Spatial Reasoning in Egocentric Video
Chaoyu Li, Arka Sadhu, Yuliang Li +3
Answering complex spatial reasoning questions from egocentric video requires determining what evidence is needed, acquiring targeted observations, and combining them through multip…
Video Generation Models: A Survey of Post-Training and Alignment
Chaoyu Li, Xiaoyi Gu, Yogesh Kulkarni +10
Video generation has rapidly progressed from short, low-quality clips to high-resolution, long-duration sequences with complex spatiotemporal dynamics. Despite strong generative pr…
Video-FLAIR: Not Whether to Reason, But How
Yogesh Kulkarni, Pooyan Fazli
Multimodal queries can require different types of reasoning. Some can be answered via perceptual reasoning, extracting information directly from the visual signal, while others req…
VisionPulse: A Virtual Reality System Enabling Accessible Discovery and Navigation for Blind and Low Vision Users
Samuel Martin, Pooyan Fazli, Hasti Seifi
Free exploration is an important aspect of many engaging virtual reality (VR) experiences, yet remains largely inaccessible to blind and low vision (BLV) users due to its reliance…
ViDscribe: Multimodal AI for Customizing Audio Description and Question Answering in Online Videos
Maryam Cheema, Sina Elahimanesh, Pooyan Fazli +1
Advances in multimodal large language models enable automatic video narration and question answering (VQA), offering scalable alternatives to labor-intensive, human-authored audio…
CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation
Chaoyu Li, Fei Tao, Pooyan Fazli
Vision-language models achieve strong performance across a wide range of multimodal understanding and reasoning tasks, yet their multi-step reasoning remains unstable. Repeated sam…
EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning
Yogesh Kulkarni, Pooyan Fazli
Egocentric video understanding requires procedural reasoning under partial observability and continuously shifting viewpoints. Current multimodal large language models (MLLMs) stru…
FrameOracle: Learning What to See and How Much to See in Videos
Chaoyu Li, Tianzhi Li, Fei Tao +6
Vision-language models (VLMs) advance video understanding but operate under tight computational budgets, making performance dependent on selecting a small, high-quality subset of f…
AVATAR: Reinforcement Learning to See, Hear, and Reason Over Video
Yogesh Kulkarni, Pooyan Fazli
Multimodal reasoning over long-horizon video is challenging due to the need for precise spatiotemporal fusion and alignment across modalities. While recent methods such as Group Re…
DescribePro: Collaborative Audio Description with Human-AI Interaction
Maryam Cheema, Sina Elahimanesh, Samuel Martin +2
Audio description (AD) makes video content accessible to millions of blind and low vision (BLV) users. However, creating high-quality AD involves a trade-off between the precision…
ReGATE: Learning Faster and Better with Fewer Tokens in MLLMs
Chaoyu Li, Yogesh Kulkarni, Pooyan Fazli
The computational cost of training multimodal large language models (MLLMs) grows rapidly with the number of processed tokens. Existing efficiency methods mainly target inference v…
VideoPASTA: 7K Preference Pairs That Matter for Video-LLM Alignment
Yogesh Kulkarni, Pooyan Fazli
Video-language models (Video-LLMs) excel at understanding video content but struggle with spatial relationships, temporal ordering, and cross-frame continuity. To address these lim…
ChartQA-X: Generating Explanations for Visual Chart Reasoning
Shamanthak Hegde, Pooyan Fazli, Hasti Seifi
The ability to explain complex information from chart images is vital for effective data-driven decision-making. In this work, we address the challenge of generating detailed expla…
VideoA11y: Method and Dataset for Accessible Video Description
Chaoyu Li, Sid Padmanabhuni, Maryam Cheema +2
Video descriptions are crucial for blind and low vision (BLV) users to access visual content. However, current artificial intelligence models for generating descriptions often fall…
VidHalluc: Evaluating Temporal Hallucinations in Multimodal Large Language Models for Video Understanding
Chaoyu Li, Eun Woo Im, Pooyan Fazli
Multimodal large language models (MLLMs) have recently shown significant advancements in video understanding, excelling in content reasoning and instruction-following tasks. Howeve…
VideoSAVi: Self-Aligned Video Language Models without Human Supervision
Yogesh Kulkarni, Pooyan Fazli
Recent advances in video-large language models (Video-LLMs) have led to significant progress in video understanding. Current preference optimization methods often rely on proprieta…
Describe Now: User-Driven Audio Description for Blind and Low Vision Individuals
Maryam Cheema, Hasti Seifi, Pooyan Fazli
Audio descriptions (AD) make videos accessible for blind and low vision (BLV) users by describing visual elements that cannot be understood from the main audio track. AD created by…
VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?
Yolo Y. Tang, Junjia Guo, Hang Hua +9
The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However…
OSCaR: Object State Captioning and State Change Representation
Nguyen Nguyen, Jing Bi, Ali Vosoughi +3
The capability of intelligent models to extrapolate and comprehend changes in object states is a crucial yet demanding aspect of AI research, particularly through the lens of human…
CLearViD: Curriculum Learning for Video Description
Cheng-Yu Chuang, Pooyan Fazli
Video description entails automatically generating coherent natural language sentences that narrate the content of a given video. We introduce CLearViD, a transformer-based model f…
Clustering Social Touch Gestures for Human-Robot Interaction
Ramzi Abou Chahine, Steven Vasquez, Pooyan Fazli +1
Social touch provides a rich non-verbal communication channel between humans and robots. Prior work has identified a set of touch gestures for human-robot interaction and described…
Charting Visual Impression of Robot Hands
Hasti Seifi, Steven A. Vasquez, Hyunyoung Kim +1
A wide variety of robotic hands have been designed to date. Yet, we do not know how users perceive these hands and feel about interacting with them. To inform hand design for socia…
NarrationBot and InfoBot: A Hybrid System for Automated Video Description
Shasta Ihorn, Yue-Ting Siu, Aditya Bodi +6
Video accessibility is crucial for blind and low vision users for equitable engagements in education, employment, and entertainment. Despite the availability of professional and am…
Real-time Policy Distillation in Deep Reinforcement Learning
Yuxiang Sun, Pooyan Fazli
Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant…