4 papers
Segment Where You Look: Attention as the Interface Between Language Models and Segmenters
Eslam Abdelrahman, Hamza Omar Alkaf, Yuheng Li +5
Vision-language models (VLMs) that segment from language must pass the location of the referent to a pixel decoder. The dominant interface compresses it into the hidden state of a…
CrossEdit: Cross-Modal Training Enables Rich Audio-Visual Editing
William Chen, Prem Seetharaman, Ke Chen +10
Multimedia editing requires generative models to interpret complex, compositional instructions and modify only the desired elements across modalities, a capability largely beyond e…
Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration
Sicheng Mo, Thao Nguyen, Richard Zhang +7
In this work, we explore an untapped signal in diffusion model inference. While all previous methods generate images independently at inference, we instead ask if samples can be ge…
X-Fusion: Introducing New Modality to Frozen Large Language Models
Sicheng Mo, Thao Nguyen, Xun Huang +9
We propose X-Fusion, a framework that extends pretrained Large Language Models (LLMs) for multimodal tasks while preserving their language capabilities. X-Fusion employs a dual-tow…