Shape-from-Template with Generalised Camera

Abstract

This article presents a new method for non-rigidly registering a 3D shape to 2D keypoints observed by a constellation of multiple cameras. Non-rigid registration of a 3D shape to observed 2D keypoints, i.e., Shape-from-Template (SfT), has been widely studied using single images, but SfT with information from multiple-cameras jointly opens new directions for extending the scope of known use-cases such as 3D shape registration in medical imaging and registration from hand-held cameras, to name a few. We represent such multi-camera setup with the generalised camera model; therefore any collection of perspective or orthographic cameras observing any deforming object can be registered. We propose multiple approaches for such SfT: the first approach where the corresponded keypoints lie on a direction vector from a known 3D point in space, the second approach where the corresponded keypoints lie on a direction vector from an unknown 3D point in space but with known orientation w.r.t some local reference frame, and a third approach where, apart from correspondences, the silhouette of the imaged object is also known. Together, these form the first set of solutions to the SfT problem with generalised cameras. The key idea behind SfT with generalised camera is the improved reconstruction accuracy from estimating deformed shape while utilising the additional information from the mutual constraints between multiple views of a deformed object. The correspondence-based approaches are solved with convex programming while the silhouette-based approach is an iterative refinement of the results from the convex solutions. We demonstrate the accuracy of our proposed methods on many synthetic and real data

Pre-print of the IMAVIS article: https://www.sciencedirect.com/science/article/abs/pii/S0262885625001672 Code and data in: https://git.zib.de/asengupta/sft-generalised

References in corpus (14)

• Point-Set Registration: Coherent Point Drift
• Manopt, a Matlab toolbox for optimization on manifolds
• XNect: Real-time Multi-Person 3D Motion Capture with a Single RGB Camera
• RoMa: Robust Dense Feature Matching
• Recovering 3D Human Mesh from Monocular Images: A Survey
• Animals in Virtual Environments
• Design and Evaluation of a Generic Visual SLAM Framework for Multi-Camera Systems
• Procrustes Analysis with Deformations: A Closed-Form Solution by Eigenvalue Decomposition
• ROBUSfT: Robust Real-Time Shape-from-Template, a C++ Library
• Automatic 3D Registration of Dental CBCT and Face Scan Data using 2D Projection Images
• Concise and Effective Network for 3D Human Modeling from Orthogonal Silhouettes
• Aligning Silhouette Topology for Self-Adaptive 3D Human Pose Recovery
• SdCT-GAN: Reconstructing CT from Biplanar X-Rays with Self-driven Generative Adversarial Networks
• Benchmarking Encoder-Decoder Architectures for Biplanar X-ray to 3D Shape Reconstruction