RGB-D-Fusion: Image Conditioned Depth Diffusion of Humanoid Subjects

Abstract

We present RGB-D-Fusion, a multi-modal conditional denoising diffusion probabilistic model to generate high resolution depth maps from low-resolution monocular RGB images of humanoid subjects. RGB-D-Fusion first generates a low-resolution depth map using an image conditioned denoising diffusion probabilistic model and then upsamples the depth map using a second denoising diffusion probabilistic model conditioned on a low-resolution RGB-D image. We further introduce a novel augmentation technique, depth noise augmentation, to increase the robustness of our super-resolution model.

References in corpus (37)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Auto-Encoding Variational Bayes
• Denoising Diffusion Probabilistic Models
• Generative Adversarial Networks
• Gaussian Error Linear Units (GELUs)
• Instance Normalization: The Missing Ingredient for Fast Stylization
• Hierarchical Text-Conditional Image Generation with CLIP Latents
• Diffusion Models Beat GANs on Image Synthesis
• Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding
• Deep Unsupervised Learning using Nonequilibrium Thermodynamics
• Score-Based Generative Modeling through Stochastic Differential Equations
• GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
• Linformer: Self-Attention with Linear Complexity
• Classifier-Free Diffusion Guidance
• DreamFusion: Text-to-3D using 2D Diffusion
• Cascaded Diffusion Models for High Fidelity Image Generation
• A note on the evaluation of generative models
• Monocular Depth Estimation Based On Deep Learning: An Overview
• Elucidating the Design Space of Diffusion-Based Generative Models
• SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations
• Variational Diffusion Models
• Confidence Propagation through CNNs for Guided Sparse Depth Regression
• Progressive Distillation for Fast Sampling of Diffusion Models
• ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth
• Point-E: A System for Generating 3D Point Clouds from Complex Prompts
• Tackling the Generative Learning Trilemma with Denoising Diffusion GANs
• Flexible Diffusion Modeling of Long Videos
• Monocular Depth Estimation: A Survey
• Novel View Synthesis with Diffusion Models
• LiDAR-guided Stereo Matching with a Spatial Consistency Constraint
• Joint Learning of Salient Object Detection, Depth Estimation and Contour Extraction
• Monocular Depth Estimation using Diffusion Models
• PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior
• Improved Vector Quantized Diffusion Models
• Multimodal Image Synthesis and Editing: The Generative AI Era
• DAG: Depth-Aware Guidance with Denoising Diffusion Probabilistic Models
• A Novel Sampling Scheme for Text- and Image-Conditional Image Synthesis in Quantized Latent Spaces