12 papers
Segment Where You Look: Attention as the Interface Between Language Models and Segmenters
Eslam Abdelrahman, Hamza Omar Alkaf, Yuheng Li +5
Vision-language models (VLMs) that segment from language must pass the location of the referent to a pixel decoder. The dominant interface compresses it into the hidden state of a…
Pristine Models
Anirudh Sundara Rajan, Krishna Kumar Singh, Richard Zhang +1
Training on broad, unfiltered data enables high-quality image generation, but can also teach models concepts that should not be generated; restricting the training data ensures saf…
Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers
Sicheng Mo, Yuheng Li, Ziyang Leng +2
Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing a…
UniTemp: Unlocking Video Generation in Any Temporal Order via Bidirectional Distillation
Lin Zhang, Sicheng Mo, Zefan Cai +6
Autoregressive video diffusion models have emerged as a promising approach for long video generation, achieving strong performance in streaming settings. However, existing methods…
Personal AI Agent for Camera Roll VQA
Thao Nguyen, Krishna Kumar Singh, Donghyun Kim +2
We study the personal camera roll visual question answering setting. In this setting, a conversational AI assistant can access a user's personal camera roll and retrieve relevant p…
MAOAM: Unified Object and Material Selection with Vision-Language Models
Jaden Park, Valentin Deschaintre, Jason Kuen +5
Selection is a core operation in interactive image editing. To be practical, a user should be able to specify and disambiguate the desired selection region through either text or c…
From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing
Anirudh Sundara Rajan, Krishna Kumar Singh, Yong Jae Lee
Modern image editing models produce realistic results but struggle with abstract, multi step instructions (e.g., ``make this advertisement more vegetarian-friendly''). Prior agent…
Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration
Sicheng Mo, Thao Nguyen, Richard Zhang +7
In this work, we explore an untapped signal in diffusion model inference. While all previous methods generate images independently at inference, we instead ask if samples can be ge…
Relational Visual Similarity
Thao Nguyen, Sicheng Mo, Krishna Kumar Singh +6
Humans do not just see attribute similarity -- we also see relational similarity. An apple is like a peach because both are reddish fruit, but the Earth is also like a peach: its c…
YoChameleon: Personalized Vision and Language Generation
Thao Nguyen, Krishna Kumar Singh, Jing Shi +3
Large Multimodal Models (e.g., GPT-4, Gemini, Chameleon) have evolved into powerful tools with millions of users. However, they remain generic models and lack personalized knowledg…
X-Fusion: Introducing New Modality to Frozen Large Language Models
Sicheng Mo, Thao Nguyen, Xun Huang +9
We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tow…
Removing Distributional Discrepancies in Captions Improves Image-Text Alignment
Yuheng Li, Haotian Liu, Mu Cai +5
In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language mo…