6 papers
PaintBench: Deterministic Evaluation of Precise Visual Editing
Kai Xu, Ellis Brown, Shrikar Madhu +3
While current multimodal models are proficient at open-ended visual editing, executing precise single-answer edits remains an important obstacle. To probe this challenge, we introd…
Beyond Language Modeling: An Exploration of Multimodal Pretraining
Shengbang Tong, David Fan, John Nguyen +18
The visual world offers a critical axis for advancing foundation models beyond language. Despite growing interest in this direction, the design space for native multimodal models r…
Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders
Shengbang Tong, Boyang Zheng, Ziteng Wang +7
Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet by training in high-dimensional semantic latent spaces. In this work, we investi…
SIMS-V: Simulated Instruction-Tuning for Spatial Video Understanding
Ellis Brown, Arijit Ray, Ranjay Krishna +3
Despite impressive high-level video comprehension, multimodal language models struggle with spatial reasoning across time and space. While current spatial training approaches rely…
Cambrian-S: Towards Spatial Supersensing in Video
Shusheng Yang, Jihan Yang, Pinzhi Huang +12
We argue that progress in true multimodal intelligence calls for a shift from reactive, task-driven systems and brute-force long context towards a broader paradigm of supersensing.…
Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts
Ellis Brown, Jihan Yang, Shusheng Yang +2
Robust benchmarks are crucial for evaluating Multimodal Large Language Models (MLLMs). Yet we find that models can ace many multimodal benchmarks without strong visual understandin…