SceneDreamer: Unbounded 3D Scene Generation from 2D Image Collections

Abstract

In this work, we present SceneDreamer, an unconditional generative model for unbounded 3D scenes, which synthesizes large-scale 3D landscapes from random noise. Our framework is learned from in-the-wild 2D image collections only, without any 3D annotations. At the core of SceneDreamer is a principled learning paradigm comprising 1) an efficient yet expressive 3D scene representation, 2) a generative scene parameterization, and 3) an effective renderer that can leverage the knowledge from 2D images. Our approach begins with an efficient bird's-eye-view (BEV) representation generated from simplex noise, which includes a height field for surface elevation and a semantic field for detailed scene semantics. This BEV scene representation enables 1) representing a 3D scene with quadratic complexity, 2) disentangled geometry and semantics, and 3) efficient training. Moreover, we propose a novel generative neural hash grid to parameterize the latent space based on 3D positions and scene semantics, aiming to encode generalizable features across various scenes. Lastly, a neural volumetric renderer, learned from 2D image collections through adversarial training, is employed to produce photorealistic images. Extensive experiments demonstrate the effectiveness of SceneDreamer and superiority over state-of-the-art methods in generating vivid yet diverse unbounded 3D worlds.

IEEE Transactions on Pattern Analysis & Machine Intelligence (TPAMI) 2023; Project Page https://scene-dreamer.github.io/ Code https://github.com/FrozenBurning/SceneDreamer

References in corpus (8)

• Instant Neural Graphics Primitives with a Multiresolution Hash Encoding
• The Replica Dataset: A Digital Replica of Indoor Spaces
• Vision Transformer Adapter for Dense Predictions
• You Only Need Adversarial Supervision for Semantic Image Synthesis
• EpiGRAF: Rethinking training of 3D GANs
• EVA3D: Compositional 3D Human Generation from 2D Image Collections
• CGOF++: Controllable 3D Face Synthesis with Conditional Generative Occupancy Fields
• Towards Better User Studies in Computer Graphics and Vision

Cited by in corpus (1)

• Make-It-4D: Synthesizing a Consistent Long-Term Dynamic Scene Video from a Single Image