Learning to Stylize Novel Views

Abstract

We tackle a 3D scene stylization problem - generating stylized images of a scene from arbitrary novel views given a set of images of the same scene and a reference image of the desired style as inputs. Direct solution of combining novel view synthesis and stylization approaches lead to results that are blurry or not consistent across different views. We propose a point cloud-based method for consistent 3D scene stylization. First, we construct the point cloud by back-projecting the image features to the 3D space. Second, we develop point cloud aggregation modules to gather the style information of the 3D scene, and then modulate the features in the point cloud with a linear transformation matrix. Finally, we project the transformed features to 2D space to obtain the novel views. Experimental results on two diverse datasets of real-world scenes validate that our method generates consistent stylized novel view synthesis results against other alternative approaches.

arXiv commentProject page: https://hhsinping.github.io/3d_scene_stylization/ Code: https://github.com/hhsinping/stylescene

References in corpus (19)

• Joint 2D-3D-Semantic Data for Indoor Scene Understanding
• Texture Networks: Feed-forward Synthesis of Textures and Stylized Images
• NeRF++: Analyzing and Improving Neural Radiance Fields
• DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks
• NeRF--: Neural Radiance Fields Without Known Camera Parameters
• Accelerating 3D Deep Learning with PyTorch3D
• Learning Linear Transformations for Fast Arbitrary Style Transfer
• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• Object-Centric Neural Scene Rendering
• ShaRF: Shape-conditioned Radiance Fields from a Single View
• Depth-supervised NeRF: Fewer Views and Faster Training for Free
• Portrait Neural Radiance Fields from a Single Image
• NeRF-VAE: A Geometry Aware 3D Scene Generative Model
• PVA: Pixel-aligned Volumetric Avatars
• PlenOctrees for Real-time Rendering of Neural Radiance Fields
• Arbitrary Video Style Transfer via Multi-Channel Correlation
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video
• FiG-NeRF: Figure-Ground Neural Radiance Fields for 3D Object Category Modelling
• Editing Conditional Radiance Fields