Neural Non-Rigid Tracking

Abstract

We introduce a novel, end-to-end learnable, differentiable non-rigid tracker that enables state-of-the-art non-rigid reconstruction by a learned robust optimization. Given two input RGB-D frames of a non-rigidly moving object, we employ a convolutional neural network to predict dense correspondences and their confidences. These correspondences are used as constraints in an as-rigid-as-possible (ARAP) optimization problem. By enabling gradient back-propagation through the weighted non-linear least squares solver, we are able to learn correspondences and confidences in an end-to-end manner such that they are optimal for the task of non-rigid tracking. Under this formulation, correspondence confidences can be learned via self-supervision, informing a learned robust optimization, where outliers and wrong correspondences are automatically down-weighted to enable effective tracking. Compared to state-of-the-art approaches, our algorithm shows improved reconstruction performance, while simultaneously achieving 85 times faster correspondence prediction than comparable deep-learning based methods. We make our code available.

arXiv commentVideo: https://youtu.be/nqYaxM6Rj8I, Code: https://github.com/DeformableFriends/NeuralTracking

References in corpus (6)

• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• BA-Net: Dense Bundle Adjustment Network
• Joint-task Self-supervised Learning for Temporal Correspondence
• RegNet: Learning the Optimization of Direct Image-to-Image Pose Registration
• Deep Learning of Preconditioners for Conjugate Gradient Solvers in Urban Water Related Problems
• Learning to Optimize Non-Rigid Tracking

Cited by in corpus (2)

• HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields
• SplitFusion: Simultaneous Tracking and Mapping for Non-Rigid Scenes