Learning from Synthetic Humans
arXiv:1701.01370 · doi:10.1109/CVPR.2017.492
Abstract
Estimating human pose, shape, and motion from images and videos are fundamental challenges with many applications. Recent advances in 2D human pose estimation use large amounts of manually-labeled training data for learning convolutional neural networks (CNNs). Such data is time consuming to acquire and difficult to extend. Moreover, manual labeling of 3D pose, depth and motion is impractical. In this work we present SURREAL (Synthetic hUmans foR REAL tasks): a new large-scale dataset with synthetically-generated but realistic images of people rendered from 3D sequences of human motion capture data. We generate more than 6 million frames together with ground truth pose, depth maps, and segmentation masks. We show that CNNs trained on our synthetic dataset allow for accurate human depth estimation and human part segmentation in real RGB images. Our results and the new dataset open up new possibilities for advancing person analysis using cheap and large-scale synthetic data.
Appears in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017). 9 pages
References in corpus (9)
- Depth Map Prediction from a Single Image using a Multi-Scale Deep Network
- LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop
- Virtual Worlds as Proxy for Multi-Object Tracking Analysis
- MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild
- Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
- Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
- Synthesizing Training Images for Boosting Human 3D Pose Estimation
- Sparseness Meets Deepness: 3D Human Pose Estimation from Monocular Video
- Learning Deep Object Detectors from 3D Models
Cited by in corpus (204)
- Embodied Hands: Modeling and Capturing Hands and Bodies Together
- Monocular Human Pose Estimation: A Survey of Deep Learning-based Methods
- LCR-Net++: Multi-person 2D and 3D Pose Detection in Natural Images
- Exploiting temporal information for 3D pose estimation
- GRAB: A Dataset of Whole-Body Human Grasping of Objects
- Unsupervised Reverse Domain Adaptation for Synthetic Medical Images via Adversarial Training
- Cascaded deep monocular 3D human pose estimation with evolutionary training data
- Recovering 3D Human Mesh from Monocular Images: A Survey
- PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization
- Self-supervised Learning of Motion Capture
- AdaFuse: Adaptive Multiview Fusion for Accurate Human Pose Estimation in the Wild
- Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks
- A survey of synthetic data augmentation methods in computer vision
- MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation
- A simple yet effective baseline for 3d human pose estimation
- Deep Learning-Based Human Pose Estimation: A Survey
- Configurable 3D Scene Synthesis and 2D Image Rendering with Per-Pixel Ground Truth using Stochastic Grammars
- Synthetic Humans for Action Recognition from Unseen Viewpoints
- SelfPose: 3D Egocentric Pose Estimation from a Headset Mounted Camera
- Cross-Domain Complementary Learning Using Pose for Multi-Person Part Segmentation
- BodyNet: Volumetric Inference of 3D Human Body Shapes
- Human Body Pose Estimation for Gait Identification: A Comprehensive Survey of Datasets and Models
- SoftSMPL: Data-driven Modeling of Nonlinear Soft-tissue Dynamics for Parametric Humans
- UnrealText: Synthesizing Realistic Scene Text Images from the Unreal World
- Towards Garment Sewing Pattern Reconstruction from a Single Image
- Style and Pose Control for Image Synthesis of Humans from a Single Monocular View
- Weakly-Supervised Discovery of Geometry-Aware Representation for 3D Human Pose Estimation
- Semi-supervised Adversarial Learning to Generate Photorealistic Face Images of New Identities from 3D Morphable Model
- Unsupervised Learning of Robust Spectral Shape Matching
- Geometric Pose Affordance: 3D Human Pose with Scene Constraints
- Deep Learning with Cinematic Rendering: Fine-Tuning Deep Neural Networks Using Photorealistic Medical Images
- Sampling Strategies for GAN Synthetic Data
- Single-shot 3D multi-person pose estimation in complex images
- When Deep Learning Meets Data Alignment: A Review on Deep Registration Networks (DRNs)
- LIGHTEN: Learning Interactions with Graph and Hierarchical TEmporal Networks for HOI in videos
- Deep Shells: Unsupervised Shape Correspondence with Optimal Transport
- Learning Multi-Human Optical Flow
- Deep learning for 3D human pose estimation and mesh recovery: A survey
- DenseBody: Directly Regressing Dense 3D Human Pose and Shape From a Single Color Image
- An Annotation Saved is an Annotation Earned: Using Fully Synthetic Training for Object Instance Detection
- Learning from THEODORE: A Synthetic Omnidirectional Top-View Indoor Dataset for Deep Transfer Learning
- Lucid Data Dreaming for Video Object Segmentation
- DCAN: Dual Channel-wise Alignment Networks for Unsupervised Scene Adaptation
- Learning to Estimate 3D Human Pose and Shape from a Single Color Image
- DeepHuman: 3D Human Reconstruction from a Single Image
- SMPLR: Deep SMPL reverse for 3D human pose and shape recovery
- Mo2Cap2: Real-time Mobile 3D Motion Capture with a Cap-mounted Fisheye Camera
- A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose
- Inference Stage Optimization for Cross-scenario 3D Human Pose Estimation
- Learning Human Optical Flow
- CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning
- InMoDeGAN: Interpretable Motion Decomposition Generative Adversarial Network for Video Generation
- Second FRCSyn-onGoing: Winning Solutions and Post-Challenge Analysis to Improve Face Recognition with Synthetic Data
- SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach
- Playing for 3D Human Recovery
- DenseRaC: Joint 3D Pose and Shape Estimation by Dense Render-and-Compare
- Learning elementary structures for 3D shape generation and matching
- Shape-Aware Human Pose and Shape Reconstruction Using Multi-View Images
- PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation
- Learning to Reconstruct People in Clothing from a Single RGB Camera
- Learning Rigidity in Dynamic Scenes with a Moving Camera for 3D Motion Field Estimation
- FBI-Pose: Towards Bridging the Gap between 2D Images and 3D Human Poses using Forward-or-Backward Information
- WheelPose: Data Synthesis Techniques to Improve Pose Estimation Performance on Wheelchair Users
- Learning Anthropometry from Rendered Humans
- Towards Robust RGB-D Human Mesh Recovery
- VersatileGait: A Large-Scale Synthetic Gait Dataset with Fine-GrainedAttributes and Complicated Scenarios
- The Whole Is Greater Than the Sum of Its Nonrigid Parts
- Sim-to-Real Learning for Casualty Detection from Ground Projected Point Cloud Data
- Detailed Human Shape Estimation from a Single Image by Hierarchical Mesh Deformation
- Semantics-Aligned Representation Learning for Person Re-identification
- Deep unsupervised 3D human body reconstruction from a sparse set of landmarks
- ElderSim: A Synthetic Data Generation Platform for Human Action Recognition in Eldercare Applications
- A Locality-based Neural Solver for Optical Motion Capture
- Shape-from-Mask: A Deep Learning Based Human Body Shape Reconstruction from Binary Mask Images
- Monocular, One-stage, Regression of Multiple 3D People
- Disentangling Geometric Deformation Spaces in Generative Latent Shape Models
- SwimXYZ: A large-scale dataset of synthetic swimming motions and videos
- Full-body High-resolution Anime Generation with Progressive Structure-conditional Generative Adversarial Networks
- GANerated Hands for Real-time 3D Hand Tracking from Monocular RGB
- Knowledge as Priors: Cross-Modal Knowledge Generalization for Datasets without Superior Knowledge
- ObjectMix: Data Augmentation by Copy-Pasting Objects in Videos for Action Recognition
- Human Image Generation: A Comprehensive Survey
- Improving 2D Human Pose Estimation in Rare Camera Views with Synthetic Data
- A Neural Anthropometer Learning from Body Dimensions Computed on Human 3D Meshes
- Learning from scarce information: using synthetic data to classify Roman fine ware pottery
- Temporally Coherent Full 3D Mesh Human Pose Recovery from Monocular Video
- RSA: Randomized Simulation as Augmentation for Robust Human Action Recognition
- HEMlets Pose: Learning Part-Centric Heatmap Triplets for Accurate 3D Human Pose Estimation
- Reconstructing Close Human Interactions from Multiple Views
- Deformation Recovery: Localized Learning for Detail-Preserving Deformations
- 3D-CODED : 3D Correspondences by Deep Deformation
- SampleAhead: Online Classifier-Sampler Communication for Learning from Synthesized Data
- Learning 3D Human Pose from Structure and Motion
- Learning Human Pose Models from Synthesized Data for Robust RGB-D Action Recognition
- BIM Hyperreality: Data Synthesis Using BIM and Hyperrealistic Rendering for Deep Learning
- Robust-DefReg: A Robust Deformable Point Cloud Registration Method based on Graph Convolutional Neural Networks
- Predicting Camera Viewpoint Improves Cross-dataset Generalization for 3D Human Pose Estimation
- 3D Generative Model Latent Disentanglement via Local Eigenprojection
- Generative Models as a Data Source for Multiview Representation Learning
- SynBench: A Synthetic Benchmark for Non-rigid 3D Point Cloud Registration
- Adversarial Appearance Learning in Augmented Cityscapes for Pedestrian Recognition in Autonomous Driving
- SynthText3D: Synthesizing Scene Text Images from 3D Virtual Worlds
- MeshAdv: Adversarial Meshes for Visual Recognition
- Computational Ceramicology
- BCNet: Learning Body and Cloth Shape from A Single Image
- Generalizing Monocular 3D Human Pose Estimation in the Wild
- Dense Pose Transfer
- Learning Continuous Environment Fields via Implicit Functions
- Local Deep Implicit Functions for 3D Shape
- HUMBI: A Large Multiview Dataset of Human Body Expressions
- Neural 3D Clothes Retargeting from a Single Image
- TetraTSDF: 3D human reconstruction from a single image with a tetrahedral outer shell
- Photo Wake-Up: 3D Character Animation from a Single Photo
- AVATAR: Blender add-on for fast creation of 3D human models
- 3D Human Pose, Shape and Texture from Low-Resolution Images and Videos
- HumanGAN: A Generative Model of Humans Images
- Latent feature disentanglement for 3D meshes
- Projection-based Classification of Surfaces for 3D Human Mesh Sequence Retrieval
- Part-level Car Parsing and Reconstruction from Single Street View
- Learning Neural Light Transport
- Rethinking Pose in 3D: Multi-stage Refinement and Recovery for Markerless Motion Capture
- CRAVES: Controlling Robotic Arm with a Vision-based Economic System
- Counting Out Time: Class Agnostic Video Repetition Counting in the Wild
- Instance-Level Task Parameters: A Robust Multi-task Weighting Framework
- Creating a Large-scale Synthetic Dataset for Human Activity Recognition
- Ordinal Depth Supervision for 3D Human Pose Estimation
- SimPose: Effectively Learning DensePose and Surface Normals of People from Simulated Data
- S3: Neural Shape, Skeleton, and Skinning Fields for 3D Human Modeling
- Learning to Generate Synthetic 3D Training Data through Hybrid Gradient
- Deformation-aware Unpaired Image Translation for Pose Estimation on Laboratory Animals
- Re-Identification Supervised Texture Generation
- EventCap: Monocular 3D Capture of High-Speed Human Motions using an Event Camera
- Unsupervised Domain Adaptation with Temporal-Consistent Self-Training for 3D Hand-Object Joint Reconstruction
- Occlusion-Invariant Rotation-Equivariant Semi-Supervised Depth Based Cross-View Gait Pose Estimation
- RoMo: A Robust Solver for Full-body Unlabeled Optical Motion Capture
- Learning from Synthetic Data for Opinion-free Blind Image Quality Assessment in the Wild
- DynaDog+T: A Parametric Animal Model for Synthetic Canine Image Generation
- Personalized 3D Human Pose and Shape Refinement
- 3D Human Texture Estimation from a Single Image with Transformers
- 3D Human Mesh Regression with Dense Correspondence
- Unsupervised Geometric Disentanglement for Surfaces via CFAN-VAE
- Pre-training without Natural Images
- Enhanced 3D Human Pose Estimation from Videos by using Attention-Based Neural Network with Dilated Convolutions
- Learning from Synthetic Animals
- Distill Knowledge from NRSfM for Weakly Supervised 3D Pose Learning
- Thinking Outside the Pool: Active Training Image Creation for Relative Attributes
- Single-view 3D Body and Cloth Reconstruction under Complex Poses
- Reconstruction of People in Loose Clothing
- Multimodal Generation of Novel Action Appearances for Synthetic-to-Real Recognition of Activities of Daily Living
- 3D Human Shape Reconstruction from a Polarization Image
- InfiniteForm: A synthetic, minimal bias dataset for fitness applications
- PanDA: Panoptic Data Augmentation
- Dual Geometric Graph Network (DG2N) -- Iterative network for deformable shape alignment
- DPC: Unsupervised Deep Point Correspondence via Cross and Self Construction
- PC-HMR: Pose Calibration for 3D Human Mesh Recovery from 2D Images/Videos
- A Neural Network for Detailed Human Depth Estimation from a Single Image
- Generating Easy-to-Understand Referring Expressions for Target Identifications
- KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes
- Regressive Domain Adaptation for Unsupervised Keypoint Detection
- PedX: Benchmark Dataset for Metric 3D Pose Estimation of Pedestrians in Complex Urban Intersections
- Kinematic-Structure-Preserved Representation for Unsupervised 3D Human Pose Estimation
- USpeech: Ultrasound-Enhanced Speech with Minimal Human Effort via Cross-Modal Synthesis
- Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors
- LBS Autoencoder: Self-supervised Fitting of Articulated Meshes to Point Clouds
- Learning Canonical Embedding for Non-rigid Shape Matching
- Neural Articulated Radiance Field
- Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation
- PoissonNet: A Local-Global Approach for Learning on Surfaces
- UltraPose: Synthesizing Dense Pose with 1 Billion Points by Human-body Decoupling 3D Model
- OperatorNet: Recovering 3D Shapes From Difference Operators
- Human Body Model Fitting by Learned Gradient Descent
- TexturePose: Supervising Human Mesh Estimation with Texture Consistency
- Multi-task human analysis in still images: 2D/3D pose, depth map, and multi-part segmentation
- Partially fake it till you make it: mixing real and fake thermal images for improved object detection
- Discovery and recognition of motion primitives in human activities
- VersatileGait: A Large-Scale Synthetic Gait Dataset Towards in-the-Wild Simulation
- Optical Mouse: 3D Mouse Pose From Single-View Video
- PoseKernelLifter: Metric Lifting of 3D Human Pose using Sound
- PeeledHuman: Robust Shape Representation for Textured 3D Human Body Reconstruction
- Joint Representation of Multiple Geometric Priors via a Shape Decomposition Model for Single Monocular 3D Pose Estimation
- Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation
- Neural Re-Rendering of Humans from a Single Image
- Robust 2D/3D Vehicle Parsing in CVIS
- Domain Adaptation for Rare Classes Augmented with Synthetic Samples
- PAUL: Procrustean Autoencoder for Unsupervised Lifting
- Learning to Train with Synthetic Humans
- Task-Assisted Domain Adaptation with Anchor Tasks
- Nuisance-Label Supervision: Robustness Improvement by Free Labels
- From Real to Synthetic and Back: Synthesizing Training Data for Multi-Person Scene Understanding
- Adversarial Domain Randomization
- Unsupervised Dense Deformation Embedding Network for Template-Free Shape Correspondence
- Pipeline for 3D reconstruction of the human body from AR/VR headset mounted egocentric cameras
- EgoRenderer: Rendering Human Avatars from Egocentric Camera Images
- 3D Human Body Reconstruction from a Single Image via Volumetric Regression
- LatentHuman: Shape-and-Pose Disentangled Latent Representation for Human Bodies
- DeepTracking-Net: 3D Tracking with Unsupervised Learning of Continuous Flow
- Inner Space Preserving Generative Pose Machine
- U-mesh: Human Correspondence Matching with Mesh Convolutional Networks
- Detailed Avatar Recovery from Single Image
- Meta Deformation Network: Meta Functionals for Shape Correspondence
- Soccer on Your Tabletop
- VolNet: Estimating Human Body Part Volumes from a Single RGB Image
- VQ-HPS: Human Pose and Shape Estimation in a Vector-Quantized Latent Space
- Shape registration in the time of transformers