15 papers
Latent-Action-Guided Video-Language Feature Learning for Surgical Instrument-Tissue Interaction Recognition
Jiajun Cheng, Subarna Tripathi, Sainan Liu +2
Recognizing instrument--tissue interactions is essential for context-aware surgical AI. Vision-language models offer a natural way to inject semantic structure into surgical repres…
Search2Motion: Training-Free Object-Level Motion Control via Attention-Consensus Search
Sainan Liu, Tz-Ying Wu, Hector A Valdez +1
We present Search2Motion, a training-free framework for object-level motion editing in image-to-video generation. Unlike prior methods requiring trajectories, bounding boxes, masks…
TrajPred: Trajectory-Conditioned Joint Embedding Prediction for Surgical Instrument-Tissue Interaction Recognition in Vision-Language Models
Jiajun Cheng, Xiaofan Yu, Subarna Tripathi +2
Recognizing instruments' interactions with tissues is essential for building context-aware AI assistants in robotic surgery. Vision-language models (VLMs) have opened a new avenue…
VC-Inspector: Advancing Reference-free Evaluation of Video Captions with Factual Analysis
Shubhashis Roy Dipta, Tz-Ying Wu, Subarna Tripathi
We propose VC-Inspector, a lightweight, open-source large multimodal model (LMM) for reference-free evaluation of video captions, with a focus on factual accuracy. Unlike existing…
Harnessing Object Grounding for Time-Sensitive Video Understanding
Tz-Ying Wu, Sharath Nittur Sridhar, Subarna Tripathi
We propose to improve the time-sensitive video understanding (TSV) capability of video large language models (Video-LLMs) with grounded objects (GO). We hypothesize that TSV tasks…
Toward Scalable Video Narration: A Training-free Approach Using Multimodal Large Language Models
Tz-Ying Wu, Tahani Trigui, Sharath Nittur Sridhar +2
In this paper, we introduce VideoNarrator, a novel training-free pipeline designed to generate dense video captions that offer a structured snapshot of video content. These caption…
EASG-Bench: Video Q&A Benchmark with Egocentric Action Scene Graphs
Ivan Rodin, Tz-Ying Wu, Kyle Min +4
We introduce EASG-Bench, a question-answering benchmark for egocentric videos where the question-answering pairs are created from spatio-temporally grounded dynamic scene graphs ca…
Keystep Recognition using Graph Neural Networks
Julia Lee Romero, Kyle Min, Subarna Tripathi +1
We pose keystep recognition as a node classification task, and propose a flexible graph-learning framework for fine-grained keystep recognition that is able to effectively leverage…
Graph-Based Multimodal and Multi-view Alignment for Keystep Recognition
Julia Lee Romero, Kyle Min, Subarna Tripathi +1
Egocentric videos capture scenes from a wearer's viewpoint, resulting in dynamic backgrounds, frequent motion, and occlusions, posing challenges to accurate keystep recognition. We…
Ego-VPA: Egocentric Video Understanding with Parameter-efficient Adaptation
Tz-Ying Wu, Kyle Min, Subarna Tripathi +1
Video understanding typically requires fine-tuning the large backbone when adapting to new domains. In this paper, we leverage the egocentric video foundation models (Ego-VFMs) bas…
SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video
Hector A. Valdez, Kyle Min, Subarna Tripathi
Pretraining egocentric vision-language models has become essential to improving downstream egocentric video-text tasks. These egocentric foundation models commonly use the transfor…
Contrastive Language Video Time Pre-training
Hengyue Liu, Kyle Min, Hector A. Valdez +1
We introduce LAVITI, a novel approach to learning language, video, and temporal representations in long-form videos via contrastive learning. Different from pre-training on video-t…
VideoSAGE: Video Summarization with Graph Representation Learning
Jose M. Rojas Chaves, Subarna Tripathi
We propose a graph-based representation learning framework for video summarization. First, we convert an input video to a graph where nodes correspond to each of the video frames.…
Action Scene Graphs for Long-Form Understanding of Egocentric Videos
Ivan Rodin, Antonino Furnari, Kyle Min +2
We present Egocentric Action Scene Graphs (EASGs), a new representation for long-form understanding of egocentric videos. EASGs extend standard manually-annotated representations o…
Dynamic Emotion Modeling with Learnable Graphs and Graph Inception Network
A. Shirian, S. Tripathi, T. Guha
Human emotion is expressed, perceived and captured using a variety of dynamic data modalities, such as speech (verbal), videos (facial expressions) and motion sensors (body gesture…