Semantic View Synthesis

Abstract

We tackle a new problem of semantic view synthesis -- generating free-viewpoint rendering of a synthesized scene using a semantic label map as input. We build upon recent advances in semantic image synthesis and view synthesis for handling photographic image content generation and view extrapolation. Direct application of existing image/view synthesis methods, however, results in severe ghosting/blurry artifacts. To address the drawbacks, we propose a two-step approach. First, we focus on synthesizing the color and depth of the visible surface of the 3D scene. We then use the synthesized color and depth to impose explicit constraints on the multiple-plane image (MPI) representation prediction process. Our method produces sharp contents at the original view and geometrically consistent renderings across novel viewpoints. The experiments on numerous indoor and outdoor images show favorable results against several strong baselines and validate the effectiveness of our approach.

arXiv commentECCV 2020. Project: https://hhsinping.github.io/svs/index.html Colab: https://colab.research.google.com/drive/1iT5PfK7zl1quAOwC227GfBjieFMVHjI5

References in corpus (6)

• GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
• Learning-Based View Synthesis for Light Field Cameras
• DRIT++: Diverse Image-to-Image Translation via Disentangled Representations
• DeepStereo: Learning to Predict New Views from the World's Imagery
• DPSNet: End-to-end Deep Plane Sweep Stereo
• Peeking Behind Objects: Layered Depth Prediction from a Single Image