27 citations · 86 across the 18 of their papers we have counts for
19 papers
Where MLLMs Fail and Why: Causal Task Decomposition for Capability Failure Diagnosis
Xia Hu, Brian Potetz, Chun-Ta Lu +6
End-to-end accuracy on compositional tasks records how often MLLMs fail, but cannot distinguish whether a failure reflects an intrinsic deficit in the targeted capability or a casc…
The Cartesian Shortcut: Re-evaluate Vision Reasoning in Polar Coordinate Space
Xia Hu, Zhenrui Yue, Brian Potetz +4
As current Multimodal Large Language Models rapidly saturate canonical visual reasoning benchmarks, a key question emerges: do these strong scores genuinely reflect robust visual u…
TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment
Bingyi Cao, Koert Chen, Kevis-Kokitsi Maninis +16
Recent progress in vision-language pretraining has enabled significant improvements to many downstream computer vision applications, such as classification, retrieval, segmentation…
Image Generators are Generalist Vision Learners
Valentin Gabeur, Shangbang Long, Songyou Peng +22
Recent works show that image and video generators exhibit zero-shot visual understanding behaviors, in a way reminiscent of how LLMs develop emergent capabilities of language under…
Life-Bench: A Benchmark and Knowledge Graph Framework for Multimodal Personalization Beyond Concept Recognition
Xia Hu, Honglei Zhuang, Brian Potetz +4
As large language models increasingly power personal assistants, users expect them to reason over multimodal life histories, from recognizing people to understanding events to aggr…
Robot Learning from a Physical World Model
Jiageng Mao, Sicheng He, Hao-Ning Wu +9
We introduce PhysWorld, a framework that enables robot learning from video generation through physical world modeling. Recent video generation models can synthesize photorealistic…
GaussianLens: Localized High-Resolution Reconstruction via On-Demand Gaussian Densification
Yijia Weng, Zhicheng Wang, Songyou Peng +3
We perceive our surroundings with an active focus, paying more attention to regions of interest, such as the shelf labels in a grocery store. When it comes to scene reconstruction,…
FirePlace: Geometric Refinements of LLM Common Sense Reasoning for 3D Object Placement
Ian Huang, Yanan Bao, Karen Truong +4
Scene generation with 3D assets presents a complex challenge, requiring both high-level semantic understanding and low-level geometric reasoning. While Multimodal Large Language Mo…
TIPS: Text-Image Pretraining with Spatial awareness
Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh +11
While image-text representation learning has become very popular in recent years, existing models tend to lack spatial awareness and have limited direct applicability for dense und…
ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images
Xiaoshuai Zhang, Zhicheng Wang, Howard Zhou +5
To advance the state of the art in the creation of 3D foundation models, this paper introduces the ConDense framework for 3D pre-training utilizing existing pre-trained 2D networks…
HAMMR: HierArchical MultiModal React agents for generic VQA
Lluis Castrejon, Thomas Mensink, Howard Zhou +3
Combining Large Language Models (LLMs) with external specialized tools (LLMs+tools) is a recent paradigm to solve multimodal tasks such as Visual Question Answering (VQA). While th…
Modeling Collaborator: Enabling Subjective Vision Classification With Minimal Human Effort via LLM Tool-Use
Imad Eddine Toubal, Aditya Avinash, Neil Gordon Alldrin +10
From content moderation to wildlife conservation, the number of applications that require models to recognize nuanced or subjective visual concepts is growing. Traditionally, devel…
Encyclopedic VQA: Visual questions about detailed properties of fine-grained categories
Thomas Mensink, Jasper Uijlings, Lluis Castrejon +6
We propose Encyclopedic-VQA, a large scale visual question answering (VQA) dataset featuring visual questions about detailed properties of fine-grained categories and instances. It…
NAVI: Category-Agnostic Image Collections with High-Quality 3D Shape and Pose Annotations
Varun Jampani, Kevis-Kokitsi Maninis, Andreas Engelhardt +13
Recent advances in neural reconstruction enable high-quality 3D object reconstruction from casually captured image collections. Current techniques mostly analyze their progress on…
LFM-3D: Learnable Feature Matching Across Wide Baselines Using 3D Signals
Arjun Karpur, Guilherme Perrotta, Ricardo Martin-Brualla +2
Finding localized correspondences across different images of the same object is crucial to understand its geometry. In recent years, this problem has seen remarkable progress with…
IBRNet: Learning Multi-View Image-Based Rendering
Qianqian Wang, Zhicheng Wang, Kyle Genova +6
We present a method that synthesizes novel views of complex scenes by interpolating a sparse set of nearby views. The core of our method is a network architecture that includes a m…
Unifying Specialist Image Embedding into Universal Image Embedding
Yang Feng, Futang Peng, Xu Zhang +7
Deep image embedding provides a way to measure the semantic similarity of two images. It plays a central role in many applications such as image search, face verification, and zero…
Blockout: Dynamic Model Selection for Hierarchical Deep Networks
Calvin Murdock, Zhen Li, Howard Zhou +1
Most deep architectures for image classification--even those that are trained to classify a large number of diverse categories--learn shared image representations with a single mod…
The Unreasonable Effectiveness of Noisy Data for Fine-Grained Recognition
Jonathan Krause, Benjamin Sapp, Andrew Howard +5
Current approaches for fine-grained recognition do the following: First, recruit experts to annotate a dataset of images, optionally also collecting more structured data in the for…