Self-Improving Visual Odometry

Abstract

We propose a self-supervised learning framework that uses unlabeled monocular video sequences to generate large-scale supervision for training a Visual Odometry (VO) frontend, a network which computes pointwise data associations across images. Our self-improving method enables a VO frontend to learn over time, unlike other VO and SLAM systems which require time-consuming hand-tuning or expensive data collection to adapt to new environments. Our proposed frontend operates on monocular images and consists of a single multi-task convolutional neural network which outputs 2D keypoints locations, keypoint descriptors, and a novel point stability score. We use the output of VO to create a self-supervised dataset of point correspondences to retrain the frontend. When trained using VO at scale on 2.5 million monocular images from ScanNet, the stability classifier automatically discovers a ranking for keypoints that are not likely to help in VO, such as t-junctions across depth discontinuities, features on shadows and highlights, and dynamic objects like people. The resulting frontend outperforms both traditional methods (SIFT, ORB, AKAZE) and deep learning methods (SuperPoint and LF-Net) in a 3D-to-2D pose estimation task on ScanNet.

References in corpus (8)

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• LF-Net: Learning Local Features from Images
• Unsupervised Learning of Depth and Ego-Motion from Video
• Deep Image Homography Estimation
• LIFT: Learned Invariant Feature Transform
• Convolutional neural network architecture for geometric matching
• FutureMapping: The Computational Structure of Spatial AI Systems
• Matching Features without Descriptors: Implicitly Matched Interest Points

Cited by in corpus (11)

• LIFT-SLAM: a deep-learning feature-based monocular visual SLAM method
• Learning Two-View Correspondences and Geometry Using Order-Aware Network
• D3VO: Deep Depth, Deep Pose and Deep Uncertainty for Monocular Visual Odometry
• Self-Supervised 3D Keypoint Learning for Ego-motion Estimation
• Neural Outlier Rejection for Self-Supervised Keypoint Learning
• Radar Odometry Combining Probabilistic Estimation and Unsupervised Feature Learning
• Advances in Inference and Representation for Simultaneous Localization and Mapping
• Unsupervised Learning of Lidar Features for Use in a Probabilistic Trajectory Estimator
• An End to End Network Architecture for Fundamental Matrix Estimation
• Incorporating Learnt Local and Global Embeddings into Monocular Visual SLAM
• Learning Local Feature Descriptor with Motion Attribute for Vision-based Localization