Discovery of Latent 3D Keypoints via End-to-end Geometric Reasoning

Abstract

This paper presents KeypointNet, an end-to-end geometric reasoning framework to learn an optimal set of category-specific 3D keypoints, along with their detectors. Given a single image, KeypointNet extracts 3D keypoints that are optimized for a downstream task. We demonstrate this framework on 3D pose estimation by proposing a differentiable objective that seeks the optimal set of keypoints for recovering the relative pose between two views of an object. Our model discovers geometrically and semantically consistent keypoints across viewing angles and instances of an object category. Importantly, we find that our end-to-end framework using no ground-truth keypoint annotations outperforms a fully supervised baseline using the same neural network architecture on the task of pose estimation. The discovered 3D keypoints on the car, chair, and plane categories of ShapeNet are visualized at http://keypointnet.github.io/.

References in corpus (9)

• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• DeepPose: Human Pose Estimation via Deep Neural Networks
• VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera
• Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation
• Learning to See by Moving
• Render for CNN: Viewpoint Estimation in Images Using CNNs Trained with Rendered 3D Model Views
• Learning to Linearize Under Uncertainty
• Adversarial Learning of Structure-Aware Fully Convolutional Networks for Landmark Localization

Cited by in corpus (44)

• Unsupervised Learning of Object Keypoints for Perception and Control
• Deep Dynamics Models for Learning Dexterous Manipulation
• Pose Estimation for Robot Manipulators via Keypoint Optimization and Sim-to-Real Transfer
• Wide-Baseline Relative Camera Pose Estimation with Directional Learning
• Supervision by Registration and Triangulation for Landmark Detection
• Causal Discovery in Physical Systems from Videos
• PVN3D: A Deep Point-wise 3D Keypoints Voting Network for 6DoF Pose Estimation
• An Analysis of SVD for Deep Rotation Estimation
• Backpropagation-Friendly Eigendecomposition
• Under the Radar: Learning to Predict Robust Keypoints for Odometry Estimation and Metric Localisation in Radar
• S3K: Self-Supervised Semantic Keypoints for Robotic Manipulation via Multi-View Consistency
• Unsupervised Learning of Probably Symmetric Deformable 3D Objects from Images in the Wild
• Accurate 6D Object Pose Estimation by Pose Conditioned Mesh Reconstruction
• UprightNet: Geometry-Aware Camera Orientation Estimation from Single Images
• Self-Supervised 3D Keypoint Learning for Ego-motion Estimation
• Self-supervised Learning of 3D Objects from Natural Images
• Semi-supervised Viewpoint Estimation with Geometry-aware Conditional Generation
• KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects
• Cerberus: A Multi-headed Derenderer
• Video Interpolation and Prediction with Unsupervised Landmarks
• Perception for Autonomous Systems (PAZ)
• End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization
• Teaching with Commentaries
• Unsupervised Learning of Landmarks based on Inter-Intra Subject Consistencies
• JOKR: Joint Keypoint Representation for Unsupervised Cross-Domain Motion Retargeting
• Vision-based system identification and 3D keypoint discovery using dynamics constraints
• Joint Learning of Semantic Alignment and Object Landmark Detection
• KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control
• Unsupervised Learning of Intrinsic Structural Representation Points
• Kit-Net: Self-Supervised Learning to Kit Novel 3D Objects into Novel 3D Cavities
• StablePose: Learning 6D Object Poses from Geometrically Stable Patches
• Photo-Geometric Autoencoding to Learn 3D Objects from Unlabelled Images
• Human Correspondence Consensus for 3D Object Semantic Understanding
• GANSeg: Learning to Segment by Unsupervised Hierarchical Image Generation
• Single Image 3D Object Estimation with Primitive Graph Networks
• ASK: Adaptively Selecting Key Local Features for RGB-D Scene Recognition
• Learning Equivariant Representations
• Fine-Grained Vehicle Perception via 3D Part-Guided Visual Data Augmentation
• StereOBJ-1M: Large-scale Stereo Image Dataset for 6D Object Pose Estimation
• Pushing it out of the Way: Interactive Visual Navigation
• SK-Net: Deep Learning on Point Cloud via End-to-end Discovery of Spatial Keypoints
• Deep Iterative Surface Normal Estimation
• Orienting Novel 3D Objects Using Self-Supervised Learning of Rotation Transforms
• ViewSynth: Learning Local Features from Depth using View Synthesis