360-Degree Panorama Generation from Few Unregistered NFoV Images

Abstract

360${}^{\circ }$ panoramas are extensively utilized as environmental light sources in computer graphics. However, capturing a 360${}^{\circ }$ $\times$ 180${}^{\circ }$ panorama poses challenges due to the necessity of specialized and costly equipment, and additional human resources. Prior studies develop various learning-based generative methods to synthesize panoramas from a single Narrow Field-of-View (NFoV) image, but they are limited in alterable input patterns, generation quality, and controllability. To address these issues, we propose a novel pipeline called PanoDiff, which efficiently generates complete 360${}^{\circ }$ panoramas using one or more unregistered NFoV images captured from arbitrary angles. Our approach has two primary components to overcome the limitations. Firstly, a two-stage angle prediction module to handle various numbers of NFoV inputs. Secondly, a novel latent diffusion-based panorama generation model uses incomplete panorama and text prompts as control signals and utilizes several geometric augmentation schemes to ensure geometric properties in generated panoramas. Experiments show that PanoDiff achieves state-of-the-art panoramic generation quality and high controllability, making it suitable for applications such as content editing.

Accepted to ACM Multimedia 2023 (MM' 23). Code is available: https://github.com/shanemankiw/Panodiff

References in corpus (11)

• Diffusion Models Beat GANs on Image Synthesis
• LAION-5B: An open large-scale dataset for training next generation image-text models
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Classifier-Free Diffusion Guidance
• Adding Conditional Control to Text-to-Image Diffusion Models
• Large Scale Image Completion via Co-Modulated Generative Adversarial Networks
• Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Model
• Deep Learning for Camera Calibration and Beyond: A Survey
• Guided Co-Modulated GAN for 360° Field of View Extrapolation
• Text2Light: Zero-Shot Text-Driven HDR Panorama Generation
• Cylin-Painting: Seamless {360\textdegree} Panoramic Image Outpainting and Beyond

Cited by in corpus (1)

• A Survey on Text-Driven 360-Degree Panorama Generation