5 papers
Semantic Action Graph: A Shared Representation for Agent Grounding and Human Interpretation of Sports Highlights
Tica Lin, Deepak Chandran, Gauri Jagatap +4
Generative agents are increasingly used to select and narrate video highlights, but they typically operate over unstructured or frame-level representations. Their output is consequ…
Video2Reaction: Training Foundation Video Models to Predict Audience Reaction
Sidong Zhang, Trang Nguyen, Shiv Shankar +4
We introduce Video2Reaction, a multimodal dataset that maps short movie segments to the induced emotional reactions of viewers in the wild, as expressed through social media commen…
DuplexJail: Spoken Interruption Attacks on Full-Duplex Speech Models
Jaechul Roh, Deepak Chandran, Amir Houmansadr +1
Full-duplex speech models accept user speech while generating responses, making input timing a potential safety concern. We introduce DuplexJail, which delivers fixed, request-inde…
Video2Reaction: Mapping Video to Audience Reaction Distribution in the Wild
Trang Nguyen, Sidong Zhang, Shiv Shankar +4
Understanding and forecasting audience reactions to video content are crucial for improving content creation, recommendation systems, and media analysis. To enable audience reactio…
A low latency attention module for streaming self-supervised speech representation learning
Jianbo Ma, Siqi Pan, Deepak Chandran +2
The transformer is a fundamental building block in deep learning, and the attention mechanism is the transformer's core component. Self-supervised speech representation learning (S…