4 papers · 1 filter
FLAT: Resampling Image and Text into 1D Flexible-Length Aligned Transmodal Tokens for Retrieval and Generation
Guangyu Sun, Shlok Kumar Mishra, Wentao Bao +8
Traditional multimodal representation learning and generation are two stages: a contrastive or self-supervised visual encoder is trained first, followed by a separate downstream ge…
RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space
Xichen Pan, Aashu Singh, Satya Narayan Shukla +3
Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative…
An Attribute-Based Measure of Video Complexity
Aditya Sarkar, Yi Li, Zihao Wang +6
A new framework for the estimation of the complexity posed by video-question pairs to video-LLMs, Video Attribute-Based Complexity (VideoABC), is proposed. Video complexity is defi…
Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generation
Chao Li, Tianhong Li, Sai Vidyaranya Nuthalapati +9
Unifying text-image contrastive learning and text-to-image (T2I) generation in a single end-to-end model is challenging because the two objectives demand opposing masking regimes:…