Publications (41)
ChartCitor: Multi-Agent Framework for Fine-Grained Chart Visual Attribution
Kanika Goswami, Puneet Mathur, Ryan Rossi +1
Large Language Models (LLMs) can perform chart question-answering tasks but often generate unverified hallucinated responses. Existing answer attribution methods struggle to ground…
An Iterative Approach for Identifying Complaint Based Tweets in Social Media Platforms
Gyanesh Anand, Akash Gautam, Puneet Mathur +3
Twitter is a social media platform where users express opinions over a variety of issues. Posts offering grievances or complaints can be utilized by private/ public organizations t…
DocSynthv2: A Practical Autoregressive Modeling for Document Generation
Sanket Biswas, Rajiv Jain, Vlad I. Morariu +5
While the generation of document layouts has been extensively explored, comprehensive document generation encompassing both layout and content presents a more complex challenge. Th…
Cluster-R1: Large Reasoning Models Are Instruction-following Clustering Agents
Peijun Qing, Puneet Mathur, Nedim Lipka +5
General-purpose embedding models excel at recognizing semantic similarities but fail to capture the characteristics of texts specified by user instructions. In contrast, instructio…
Blind to the Human Touch: Overlap Bias in LLM-Based Summary Evaluation
Jiangnan Fang, Cheng-Tse Liu, Hanieh Deilamsalehy +5
Large language model (LLM) judges have often been used alongside traditional, algorithm-based metrics for tasks like summarization because they better capture semantic information,…
Estimating Emotion Contagion on Social Media via Localized Diffusion in Dynamic Graphs
Trisha Mittal, Puneet Mathur, Rohan Chandra +5
We present a computational approach for estimating emotion contagion on social media networks. Built on a foundation of psychology literature, our approach estimates the degree to…
DynaSaur: Large Language Agents Beyond Predefined Actions
Dang Nguyen, Viet Dac Lai, Seunghyun Yoon +9
Existing LLM agent systems typically select actions from a fixed and predefined set at every step. While this approach is effective in closed, narrowly scoped environments, it pres…
Lizard: An Efficient Linearization Framework for Large Language Models
Chien Van Nguyen, Huy Nguyen, Ruiyi Zhang +10
We propose Lizard, a linearization framework that transforms pretrained Transformer-based Large Language Models (LLMs) into subquadratic architectures. Transformers faces severe co…
3MASSIV: Multilingual, Multimodal and Multi-Aspect dataset of Social Media Short Videos
Vikram Gupta, Trisha Mittal, Puneet Mathur +5
We present 3MASSIV, a multilingual, multimodal and multi-aspect, expertly-annotated dataset of diverse short videos extracted from short-video social media platform - Moj. 3MASSIV…
A Survey on LLM-based Conversational User Simulation
Bo Ni, Leyao Wang, Yu Wang +27
User simulation has long played a vital role in computer science due to its potential to support a wide range of applications. Language, as the primary medium of human communicatio…
Charts Are Not Images: On the Challenges of Scientific Chart Editing
Shawn Li, Ryan Rossi, Sungchul Kim +5
Generative models, such as diffusion and autoregressive approaches, have demonstrated impressive capabilities in editing natural images. However, applying these tools to scientific…
Survey of User Interface Design and Interaction Techniques in Generative AI Applications
Reuben Luera, Ryan A. Rossi, Alexa Siu +10
The applications of generative AI have become extremely impressive, and the interplay between users and AI is even more so. Current human-AI interaction literature has taken a broa…
Follow the Flow: Fine-grained Flowchart Attribution with Neurosymbolic Agents
Manan Suri, Puneet Mathur, Nedim Lipka +4
Flowcharts are a critical tool for visualizing decision-making processes. However, their non-linear structure and complex visual-textual relationships make it challenging to interp…
Multi-LLM Text Summarization
Jiangnan Fang, Cheng-Tse Liu, Jieun Kim +9
In this work, we propose a Multi-LLM summarization framework, and investigate two different multi-LLM strategies including centralized and decentralized. Our multi-LLM summarizatio…
GraphicBench: A Planning Benchmark for Graphic Design with Language Agents
Dayeon Ki, Tianyi Zhou, Marine Carpuat +3
Large Language Model (LLM)-powered agents have unlocked new possibilities for automating human tasks. While prior work has focused on well-defined tasks with specified goals, the c…
#MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement
Akash Gautam, Puneet Mathur, Rakesh Gosangi +3
In this paper, we present a dataset containing 9,973 tweets related to the MeToo movement that were manually annotated for five different linguistic aspects: relevance, stance, hat…
SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps
Neha Srikanth, Victor Bursztyn, Puneet Mathur +1
We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of…
MODS: Moderating a Mixture of Document Speakers to Summarize Debatable Queries in Document Collections
Nishant Balepur, Alexa Siu, Nedim Lipka +4
Query-focused summarization (QFS) gives a summary of documents to answer a query. Past QFS work assumes queries have one answer, ignoring debatable ones (Is law school worth it?).…
Reasoning-Based Personalized Generation for Users with Sparse Data
Bo Ni, Branislav Kveton, Samyadeep Basu +14
Large Language Model (LLM) personalization holds great promise for tailoring responses by leveraging personal context and history. However, real-world users usually possess sparse…
Taipan: Efficient and Expressive State Space Language Models with Selective Attention
Chien Van Nguyen, Huy Huu Nguyen, Thang M. Pham +8
Efficient long-context language modeling remains a significant challenge in Natural Language Processing (NLP). While Transformers dominate language tasks, they struggle with long s…
Dynamic Graph Modeling of Simultaneous EEG and Eye-tracking Data for Reading Task Identification
Puneet Mathur, Trisha Mittal, Dinesh Manocha
We present a new approach, that we call AdaGTCN, for identifying human reader intent from Electroencephalogram~(EEG) and Eye movement~(EM) data in order to help differentiate betwe…
Test-Time Strategies for More Efficient and Accurate Agentic RAG
Brian Zhang, Deepti Guntur, Zhiyang Zuo +7
Retrieval-Augmented Generation (RAG) systems face challenges with complex, multihop questions, and agentic frameworks such as Search-R1 (Jin et al., 2025), which operates iterative…
PlotGen: Multi-Agent LLM-based Scientific Data Visualization via Multimodal Feedback
Kanika Goswami, Puneet Mathur, Ryan Rossi +1
Scientific data visualization is pivotal for transforming raw data into comprehensible visual representations, enabling pattern recognition, forecasting, and the presentation of da…
DocEdit-v2: Document Structure Editing Via Multimodal LLM Grounding
Manan Suri, Puneet Mathur, Franck Dernoncourt +5
Document structure editing involves manipulating localized textual, visual, and layout components in document images based on the user's requests. Past works have shown that multim…
PlotEdit: Natural Language-Driven Accessible Chart Editing in PDFs via Multimodal LLM Agents
Kanika Goswami, Puneet Mathur, Ryan Rossi +1
Chart visualizations, while essential for data interpretation and communication, are predominantly accessible only as images in PDFs, lacking source data tables and stylistic infor…
Structured Uncertainty guided Clarification for LLM Agents
Manan Suri, Puneet Mathur, Nedim Lipka +3
LLM agents with tool-calling capabilities often fail when user instructions are ambiguous or incomplete, leading to incorrect invocations and task failures. Existing approaches ope…
InfinityStory: Unlimited Video Generation with World Consistency and Character-Aware Shot Transitions
Mohamed Elmoghany, Liangbing Zhao, Xiaoqian Shen +27
Generating long-form storytelling videos with consistent visual narratives remains a significant challenge in video synthesis. We present a novel framework, dataset, and a model th…
GUI Agents: A Survey
Dang Nguyen, Jian Chen, Yu Wang +27
Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomo…
DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Gaurav Najpande +4
Diagram question answering (DQA) requires models to interpret structured visual representations such as charts, maps, infographics, circuit schematics, and scientific diagrams. Rec…
A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality
Mohamed Elmoghany, Ryan Rossi, Seunghyun Yoon +26
Despite the significant progress that has been made in video generative models, existing state-of-the-art methods can only produce videos lasting 5-16 seconds, often labeled "long-…
Affect2MM: Affective Analysis of Multimedia Content Using Emotion Causality
Trisha Mittal, Puneet Mathur, Aniket Bera +1
We present Affect2MM, a learning method for time-series emotion prediction for multimedia content. Our goal is to automatically capture the varying emotions depicted by characters…
FigCaps-HF: A Figure-to-Caption Generative Framework and Benchmark with Human Feedback
Ashish Singh, Ashutosh Singh, Prateek Agarwal +10
Captions are crucial for understanding scientific visualizations and documents. Existing captioning methods for scientific figures rely on figure-caption pairs extracted from docum…
MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces
Reuben A. Luera, Ryan Rossi, Franck Dernoncourt +12
In an ideal design pipeline, user interface (UI) design is intertwined with user research to validate decisions, yet studies are often resource-constrained during early exploration…
From Selection to Generation: A Survey of LLM-based Active Learning
Yu Xia, Subhojyoti Mukherjee, Zhouhang Xie +31
Active Learning (AL) has been a powerful paradigm for improving model efficiency and performance by selecting the most informative data points for labeling and training. In recent…
Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models
Puneet Mathur, Manan Suri, Dinesh Manocha
Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent toke…
Partial Policy Gradients for RL in LLMs
Puneet Mathur, Branislav Kveton, Subhojyoti Mukherjee +1
Reinforcement learning is a framework for learning to act sequentially in an unknown environment. We propose a natural approach for modeling policy structure in policy gradients. T…
Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
Runzhou Liu, Hailey Weingord, Sejal Mittal +18
Evaluating image editing models remains challenging due to the coarse granularity and limited interpretability of traditional metrics, which often fail to capture aspects important…
DIAGRAMS: A Review Framework for Reasoning-Level Attribution in Diagram QA
Anirudh Iyengar Kaniyar Narayana Iyengar, Tampu Ravi Kumar, Manan Suri +4
Diagram question answering (Diagram QA) requires reasoning-level attribution that links each question-answer pair to all visual regions needed to derive the answer, rather than onl…
ChartLens: Fine-grained Visual Attribution in Charts
Manan Suri, Puneet Mathur, Nedim Lipka +3
The growing capabilities of multimodal large language models (MLLMs) have advanced tasks like chart understanding. However, these models often suffer from hallucinations, where gen…
Sparse Personalized Text Generation with Multi-Trajectory Reasoning
Bo Ni, Haowei Fu, Qinwen Ge +10
As Large Language Models (LLMs) advance, personalization has become a key mechanism for tailoring outputs to individual user needs. However, most existing methods rely heavily on d…
VisDoM: Multi-Document QA with Visually Rich Elements Using Multimodal Retrieval-Augmented Generation
Manan Suri, Puneet Mathur, Franck Dernoncourt +3
Understanding information from a collection of multiple documents, particularly those with visually rich elements, is important for document-grounded question answering. This paper…