18 papers
MultiPartyTurnBench: Benchmarking Floor Control in Multi-Party Spoken Dialogue
Puneet Mathur, Dinesh Manocha
Full-duplex speech systems must decide when to enter, hold, or relinquish the conversational floor, even as multiple speakers overlap or backchannel without taking it. Existing eva…
DuplexSpeechBench-Document Grounding: Benchmarking Document Grounding and Hallucinations in Voice Agents
Puneet Mathur, Nedim Lipka, Zeyu Jin +1
Voice agents enable low-latency, natural interaction, yet their ability to faithfully ground responses in external documents remains underexplored. We introduce DuplexSpeechBench-D…
APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction
Puneet Mathur, Dinesh Manocha
Full-duplex voice agents can now listen, speak, use tools, and act during spoken interactions, but fluent dialogue does not guarantee correct completion of delegated professional w…
ASSEMBLE: Atomic Skills for Evidence-Grounded Video Reasoning
Xiyang Wu, Zongxia Li, Shengxin Zhang +2
Complex video reasoning often depends on evidence scattered across distant moments, entities, and events, yet a correct answer alone does not reveal whether a model relied on the r…
FD-VAD: Semantic Endpoint Detection for Streaming Full-Duplex Speech
Puneet Mathur, Dinesh Manocha
Natural turn-taking in full-duplex voice interaction requires determining from partial speech whether a pause reflects hesitation or a completed conversational intent. Acoustic voi…
DuplexSpeechBench-IFEval: Evaluating Implicit Instruction Following in Full-Duplex Voice Agents
Puneet Mathur, Dinesh Manocha
Full-duplex voice agents must continuously decide when to speak, listen, backchannel, interrupt, overlap, and yield the conversational floor. Existing benchmarks evaluate these beh…
Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models
Puneet Mathur, Manan Suri, Dinesh Manocha
Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent toke…
Frames2LoRA: Parametric Video Internalization for Vision-Language Models
Manan Suri, Sarvesh Baskar, Dinesh Manocha
Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce F…
DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Manan Suri +4
Diagram question answering (Diagram QA) requires reasoning-level attribution that links each question-answer pair to all visual regions needed to derive the answer, rather than onl…
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4
Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Rec…
Learning Illumination Control in Diffusion Models
Nishit Anand, Manan Suri, Christopher Metzler +2
Controlling illumination in images is essential for photography and visual content creation. While closed-source models have demonstrated impressive illumination control, open-sour…
InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions
Mohamed Elmoghany, Liangbing Zhao, Xiaoqian Shen +27
Generating long-form storytelling videos with consistent visual narratives remains a significant challenge in video synthesis. We present a novel framework, dataset, and a model th…
Structured Uncertainty guided Clarification for LLM Agents
Manan Suri, Puneet Mathur, Nedim Lipka +3
LLM agents with tool-calling capabilities often fail when user instructions are ambiguous or incomplete, leading to incorrect invocations and task failures. Existing approaches ope…
A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality
Mohamed Elmoghany, Ryan Rossi, Seunghyun Yoon +26
Despite the significant progress that has been made in video generative models, existing state-of-the-art methods can only produce videos lasting 5-16 seconds, often labeled "long-…
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
Manan Suri, Puneet Mathur, Nedim Lipka +4
Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interp…
ChartLens: Fine-grained Visual Attribution in Charts
Manan Suri, Puneet Mathur, Nedim Lipka +3
The growing capabilities of multimodal large language models (MLLMs) have advanced tasks like chart understanding. However, these models often suffer from hallucinations, where gen…
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
Manan Suri, Puneet Mathur, Franck Dernoncourt +3
Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper…
DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding
Manan Suri, Puneet Mathur, Franck Dernoncourt +5
Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multim…