128 citations · 251 across the 37 of their papers we have counts for
41 papers
SpatialStack: Layered Geometry-Language Fusion for 3D VLM Spatial Reasoning
Jian Zhang, Shijie Zhou, Bangya Liu +2
Large vision-language models (VLMs) still struggle with reliable 3D spatial reasoning, a core capability for embodied and physical AI systems. This limitation arises from their ina…
MoCA3D: Monocular 3D Bounding Box Prediction in the Image Plane
Changwoo Jeon, Rishi Upadhyay, Achuta Kadambi
Monocular 3D object understanding has largely been cast as a 2D RoI-to-3D box lifting problem. However, emerging downstream applications require image-plane geometry (e.g., project…
WorldBench: Benchmarking Physical Understanding of World Models by Isolating Physics Concepts
Rishi Upadhyay, Howard Zhang, Jim Solomon +5
Recent advances in generative foundational models, often termed "world models," have propelled interest in applying them to critical tasks like robotic planning and autonomous syst…
MorphoSim: An Interactive, Controllable, and Editable Language-guided 4D World Simulator
Xuehai He, Shijie Zhou, Thivyanth Venkateswaran +4
World models that support controllable and editable spatiotemporal environments are valuable for robotics, enabling scalable training data, repro ducible evaluation, and flexible t…
VLM4D: Towards Spatiotemporal Awareness in Vision Language Models
Shijie Zhou, Alexander Vilesov, Xuehai He +7
Vision language models (VLMs) have shown remarkable capabilities in integrating linguistic and visual reasoning but remain fundamentally limited in understanding dynamic spatiotemp…
Feature4X: Bridging Any Monocular Video to 4D Agentic AI with Versatile Gaussian Feature Fields
Shijie Zhou, Hui Ren, Yijia Weng +8
Recent advancements in 2D and multimodal models have achieved remarkable success by leveraging large-scale training on extensive datasets. However, extending these achievements to…
InstantRestore: Single-Step Personalized Face Restoration with Shared-Image Attention
Howard Zhang, Yuval Alaluf, Sizhuo Ma +3
Face image restoration aims to enhance degraded facial images while addressing challenges such as diverse degradation types, real-time processing demands, and, most crucially, the…
2-Factor Retrieval for Improved Human-AI Decision Making in Radiology
Jim Solomon, Laleh Jalilian, Alexander Vilesov +4
Human-machine teaming in medical AI requires us to understand to what degree a trained clinician should weigh AI predictions. While previous work has shown the potential of AI assi…
Large Spatial Model: End-to-end Unposed Images to Semantic 3D
Zhiwen Fan, Jian Zhang, Wenyan Cong +10
Reconstructing and understanding 3D structures from a limited number of images is a well-established problem in computer vision. Traditional methods usually break this task into mu…
Thermal Imaging and Radar for Remote Sleep Monitoring of Breathing and Apnea
Kai Del Regno, Alexander Vilesov, Adnan Armouti +4
Polysomnography (PSG), the current gold standard method for monitoring and detecting sleep disorders, is cumbersome and costly. At-home testing solutions, known as home sleep apnea…
Solutions to Deepfakes: Can Camera Hardware, Cryptography, and Deep Learning Verify Real Images?
Alexander Vilesov, Yuan Tian, Nader Sehatbakhsh +1
The exponential progress in generative AI poses serious implications for the credibility of all real images and videos. There will exist a point in the future where 1) digital cont…
The Potential and Perils of Generative Artificial Intelligence for Quality Improvement and Patient Safety
Laleh Jalilian, Daniel McDuff, Achuta Kadambi
Generative artificial intelligence (GenAI) has the potential to improve healthcare through automation that enhances the quality and safety of patient care. Powered by foundation mo…
4K4DGen: Panoramic 4D Generation at 4K Resolution
Renjie Li, Panwang Pan, Bangbang Yang +8
The blooming of virtual reality and augmented reality (VR/AR) technologies has driven an increasing demand for the creation of high-quality, immersive, and dynamic environments. Ho…
All-day Depth Completion
Vadim Ezhov, Hyoungseob Park, Zhaoyang Zhang +7
We propose a method for depth estimation under different illumination conditions, i.e., day and night time. As photometry is uninformative in regions under low-illumination, we tac…
DreamScene360: Unconstrained Text-to-3D Scene Generation with Panoramic Gaussian Splatting
Shijie Zhou, Zhiwen Fan, Dejia Xu +6
The increasing demand for virtual reality applications has highlighted the significance of crafting immersive 3D assets. We present a text-to-3D 360 scene generation pipe…
GT-Rain Single Image Deraining Challenge Report
Howard Zhang, Yunhao Ba, Ethan Yang +20
This report reviews the results of the GT-Rain challenge on single image deraining at the UG2+ workshop at CVPR 2023. The aim of this competition is to study the rainy weather phen…
WeatherProof: Leveraging Language Guidance for Semantic Segmentation in Adverse Weather
Blake Gella, Howard Zhang, Rishi Upadhyay +7
We propose a method to infer semantic segmentation maps from images captured under adverse weather conditions. We begin by examining existing models on images degraded by weather c…
Personalized Restoration via Dual-Pivot Tuning
Pradyumna Chari, Sizhuo Ma, Daniil Ostashev +4
Generative diffusion models can serve as a prior which ensures that solutions of image restoration systems adhere to the manifold of natural images. However, for restoring facial i…
MVDD: Multi-View Depth Diffusion Models
Zhen Wang, Qiangeng Xu, Feitong Tan +6
Denoising diffusion models have demonstrated outstanding results in 2D image generation, yet it remains a challenge to replicate its success in 3D shape generation. In this paper,…
WeatherProof: A Paired-Dataset Approach to Semantic Segmentation in Adverse Weather
Blake Gella, Howard Zhang, Rishi Upadhyay +5
The introduction of large, foundational models to computer vision has led to drastically improved performance on the task of semantic segmentation. However, these existing methods…
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields
Shijie Zhou, Haoran Chang, Sicheng Jiang +7
3D scene representations have gained immense popularity in recent years. Methods that use Neural Radiance fields are versatile for traditional tasks such as novel view synthesis. I…
Enhancing Diffusion Models with 3D Perspective Geometry Constraints
Rishi Upadhyay, Howard Zhang, Yunhao Ba +5
While perspective is a well-studied topic in art, it is generally taken for granted in images. However, for the recent wave of high-quality image synthesis methods such as latent d…
SparseGS: Sparse View Synthesis using 3D Gaussian Splatting
Haolin Xiong, Sairisheek Muttukuru, Hanyuan Xiao +4
3D Gaussian Splatting (3DGS) has recently enabled real-time rendering of unbounded 3D scenes for novel view synthesis. However, this technique requires dense training views to accu…
CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting
Alexander Vilesov, Pradyumna Chari, Achuta Kadambi
With the onset of diffusion-based generative models and their ability to generate text-conditioned images, content generation has received a massive invigoration. Recently, these m…