4 papers
Is CLIP ideal? No. Can we fix it? Yes!
Raphi Kang, Yue Song, Georgia Gkioxari +1
Contrastive Language-Image Pre-Training (CLIP) is a popular method for learning multimodal latent spaces with well-organized semantics. Despite its wide range of applications, CLIP…
Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models
Raphi Kang, Hongqiao Chen, Georgia Gkioxari +1
Spatio-temporal reasoning is a remarkable capability of Vision Language Models (VLMs), but the underlying mechanisms of such abilities remain largely opaque. We postulate that visu…
NitroGen: An Open Foundation Model for Generalist Gaming Agents
Loïc Magne, Anas Awadalla, Guanzhi Wang +11
We introduce NitroGen, a vision-action foundation model for generalist gaming agents that is trained on 40,000 hours of gameplay videos across more than 1,000 games. We incorporate…
Feedforward 3D Editing via Text-Steerable Image-to-3D
Ziqi Ma, Hongqiao Chen, Yisong Yue +1
Recent progress in image-to-3D has opened up immense possibilities for design, AR/VR, and robotics. However, to use AI-generated 3D assets in real applications, a critical requirem…