HOT-POT: Optimal Transport for Sparse Stereo Matching

Abstract

Stereo vision between images faces a range of challenges, including occlusions, motion, and camera distortions, across applications in autonomous driving, robotics, and face analysis. Due to parameter sensitivity, further complications arise for stereo matching with sparse features, such as facial landmarks. To overcome this ill-posedness and enable unsupervised sparse matching, we consider line constraints of the camera geometry from an optimal transport (OT) viewpoint. Formulating camera-projected points as (half)lines, we propose the use of the classical epipolar distance as well as a 3D ray distance to quantify matching quality. Employing these distances as a cost function of a (partial) OT problem, we arrive at efficiently solvable assignment problems. Moreover, we extend our approach to unsupervised object matching by formulating it as a hierarchical OT problem. The resulting algorithms allow for efficient feature and object matching, as demonstrated in our numerical experiments. Here, we focus on applications in facial analysis, where we aim to match distinct landmarking conventions.

14 pages, 9 figures, 8 tables

References in corpus (7)

• How far are we from solving the 2D & 3D Face Alignment problem? (and a dataset of 230,000 3D facial landmarks)
• Facial Landmark Detection: a Literature Survey
• Unbalanced Multi-Marginal Optimal Transport
• Lagrangian Motion Magnification with Double Sparse Optical Flow Decomposition
• Accelerating the Sinkhorn algorithm for sparse multi-marginal optimal transport by fast Fourier transforms
• Sliced Optimal Transport on the Sphere
• Parallelly Sliced Optimal Transport on Spheres and on the Rotation Group