PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization

Abstract

We introduce Pixel-aligned Implicit Function (PIFu), a highly effective implicit representation that locally aligns pixels of 2D images with the global context of their corresponding 3D object. Using PIFu, we propose an end-to-end deep learning method for digitizing highly detailed clothed humans that can infer both 3D surface and texture from a single image, and optionally, multiple input images. Highly intricate shapes, such as hairstyles, clothing, as well as their variations and deformations can be digitized in a unified way. Compared to existing representations used for 3D deep learning, PIFu can produce high-resolution surfaces including largely unseen regions such as the back of a person. In particular, it is memory efficient unlike the voxel representation, can handle arbitrary topology, and the resulting surface is spatially aligned with the input image. Furthermore, while previous techniques are designed to process either a single image or multiple views, PIFu extends naturally to arbitrary number of views. We demonstrate high-resolution and robust reconstructions on real world images from the DeepFashion dataset, which contains a variety of challenging clothing types. Our method achieves state-of-the-art performance on a public benchmark and outperforms the prior work for clothed human digitization from a single image.

project page: https://shunsukesaito.github.io/PIFu

References in corpus (7)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Learning from Synthetic Humans
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• PixelNet: Representation of the pixels, by the pixels, and for the pixels
• Towards Accurate Markerless Human Shape and Pose Estimation over Time
• Hierarchical Surface Prediction for 3D Object Reconstruction
• Photo Wake-Up: 3D Character Animation from a Single Photo

Cited by in corpus (37)

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains
• GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis
• Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance
• Differentiable Rendering: A Survey
• NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction
• CIPS-3D: A 3D-Aware Generator of GANs Based on Conditionally-Independent Pixel Synthesis
• Inverse Graphics GAN: Learning to Generate 3D Shapes from Unstructured 2D Data
• Generative Adversarial Networks for Image and Video Synthesis: Algorithms and Applications
• MulayCap: Multi-layer Human Performance Capture Using A Monocular Video Camera
• IBRNet: Learning Multi-View Image-Based Rendering
• PVA: Pixel-aligned Volumetric Avatars
• DeepCap: Monocular Human Performance Capture Using Weak Supervision
• Deep Fashion3D: A Dataset and Benchmark for 3D Garment Reconstruction from Single Images
• Convolutional Occupancy Networks
• Neural Sparse Voxel Fields
• Reproducing Activation Function for Deep Learning
• BCNet: Learning Body and Cloth Shape from A Single Image
• HUMBI: A Large Multiview Dataset of Human Body Expressions
• SALD: Sign Agnostic Learning with Derivatives
• Fine Structure-Aware Sampling: A New Sampling Training Scheme for Pixel-Aligned Implicit Models in Single-View Human Reconstruction
• 3D Human Shape and Pose from a Single Low-Resolution Image with Self-Supervised Learning
• MonoClothCap: Towards Temporally Coherent Clothing Capture from Monocular RGB Video
• Learning Implicit Functions for Topology-Varying Dense 3D Shape Correspondence
• 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
• Monocular Real-Time Volumetric Performance Capture
• NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenes
• Efficient Realistic Data Generation Framework leveraging Deep Learning-based Human Digitization
• SPSG: Self-Supervised Photometric Scene Generation from RGB-D Scans
• Ray-ONet: Efficient 3D Reconstruction From A Single RGB Image
• Flow Guided Transformable Bottleneck Networks for Motion Retargeting
• Neural Rays for Occlusion-aware Image-based Rendering
• SSN: Soft Shadow Network for Image Compositing
• ImpliCity: City Modeling from Satellite Images with Deep Implicit Occupancy Fields
• Toward Accurate and Realistic Outfits Visualization with Attention to Details
• Fully Understanding Generic Objects: Modeling, Segmentation, and Reconstruction
• PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction
• Robust 3D Self-portraits in Seconds