12 papers
Learning to Refer from Estimated Listener Gaze
Téa Wright, Alane Suhr
We propose to finetune vision-language models to generate more pragmatically optimal referring expressions by transforming observations of incremental listener comprehension, in th…
Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation
Anya Ji, Abhijith Varma Mudunuri, David M. Chan +1
While recent vision-language models (VLMs) have achieved significant improvements on static visual-to-code tasks such as generating code for webpages, charts, or SVGs, it remains u…
Representational Similarity and Model Behavior in Multi-Agent Interaction
Yujin Potter, Seun Eisape, Shiyang Lai +6
Researchers have shown that neural similarity among humans predicts social closeness and cooperative success, whereas innovation often emerges from interactions among dissimilar in…
DigitalCoach: Communication and Grounding Gaps in Human and Agentic Computer Use Coaching
Meng Chen, Anya Ji, Tsung-Han Wu +4
Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach, a multimodal dataset of 72 human…
ScribbleEdit: Synthetic Data for Image Editing with Scribbles and Text
Anya Ji, George Ma, Téa Wright +4
Recent progress in generative models has significantly advanced image editing capabilities, yet precise and intuitive user control remains difficult. Specifically, users often stru…
Characterizing Language Use in a Collaborative Situated Game
Nicholas Tomlin, Naitian Zhou, Eve Fleisig +10
Cooperative video games, where multiple participants must coordinate by communicating and reasoning under uncertainty in complex environments, yield a rich source of language data.…
Long Chain-of-Thought Reasoning Across Languages
Josh Barua, Seun Eisape, Kayo Yin +1
While large reasoning models have shown remarkable ability to generate long chains-of-thought (CoTs) in English, we still lack understanding of how these long-form reasoning abilit…
TULIP: Towards Unified Language-Image Pretraining
Zineng Tang, Long Lian, Seun Eisape +6
Despite the recent success of image-text contrastive models like CLIP and SigLIP, these models often struggle with vision-centric tasks that demand high-fidelity image understandin…
AutoPresent: Designing Structured Visuals from Scratch
Jiaxin Ge, Zora Zhiruo Wang, Xuhui Zhou +8
Designing structured visuals such as presentation slides is essential for communicative needs, necessitating both content creation and visual planning skills. In this work, we tack…
Evaluating Model Perception of Color Illusions in Photorealistic Scenes
Lingjun Mao, Zineng Tang, Alane Suhr
We study the perception of color illusions by vision-language models. Color illusion, where a person's visual system perceives color differently from actual color, is well-studied…
Using Language Models to Disambiguate Lexical Choices in Translation
Josh Barua, Sanjay Subramanian, Kayo Yin +1
In translation, a concept represented by a single word in a source language can have multiple variations in a target language. The task of lexical selection requires using context…
Grounding Language in Multi-Perspective Referential Communication
Zineng Tang, Lingjun Mao, Alane Suhr
We introduce a task and dataset for referring expression generation and comprehension in multi-agent embodied environments. In this task, two agents in a shared scene must take int…