Fusing Structure from Motion and Simulation-Augmented Pose Regression from Optical Flow for Challenging Indoor Environments

Abstract

The localization of objects is a crucial task in various applications such as robotics, virtual and augmented reality, and the transportation of goods in warehouses. Recent advances in deep learning have enabled the localization using monocular visual cameras. While structure from motion (SfM) predicts the absolute pose from a point cloud, absolute pose regression (APR) methods learn a semantic understanding of the environment through neural networks. However, both fields face challenges caused by the environment such as motion blur, lighting changes, repetitive patterns, and feature-less structures. This study aims to address these challenges by incorporating additional information and regularizing the absolute pose using relative pose regression (RPR) methods. RPR methods suffer under different challenges, i.e., motion blur. The optical flow between consecutive images is computed using the Lucas-Kanade algorithm, and the relative pose is predicted using an auxiliary small recurrent convolutional network. The fusion of absolute and relative poses is a complex task due to the mismatch between the global and local coordinate systems. State-of-the-art methods fusing absolute and relative poses use pose graph optimization (PGO) to regularize the absolute pose predictions using relative poses. In this work, we propose recurrent fusion networks to optimally align absolute and relative pose predictions to improve the absolute pose prediction. We evaluate eight different recurrent units and construct a simulation environment to pre-train the APR and RPR networks for better generalized training. Additionally, we record a large database of different scenarios in a challenging large-scale indoor environment that mimics a warehouse with transportation robots. We conduct hyperparameter searches and experiments to show the effectiveness of our recurrent fusion method compared to PGO.

References in corpus (22)

• Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
• An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling
• LSTM Fully Convolutional Networks for Time Series Classification
• Multivariate LSTM-FCNs for Time Series Classification
• DeepVO: Towards End-to-End Visual Odometry with Deep Recurrent Convolutional Neural Networks
• Direct Sparse Visual-Inertial Odometry using Dynamic Marginalization
• VLocNet++: Deep Multitask Learning for Semantic Visual Localization and Odometry
• Deep Gated Recurrent and Convolutional Network Hybrid Model for Univariate Time Series Classification
• LS-VO: Learning Dense Optical Subspace for Robust Visual Odometry Estimation
• What is the best RNN-cell structure to forecast each time series behavior?
• Sequencer: Deep LSTM for Image Classification
• XceptionTime: A Novel Deep Architecture based on Depthwise Separable Convolutions for Hand Gesture Classification
• ViPR: Visual-Odometry-aided Pose Regression for 6DoF Camera Localization
• Monocular Depth Estimation using Diffusion Models
• Homography-Based Loss Function for Camera Pose Regression
• Unsupervised Joint Learning of Depth, Optical Flow, Ego-motion from Video
• Benchmarking Visual-Inertial Deep Multimodal Fusion for Relative Pose Regression and Odometry-aided Absolute Pose Regression
• CGA-PoseNet: Camera Pose Regression via a 1D-Up Approach to Conformal Geometric Algebra
• Improving the generalization of network based relative pose regression: dimension reduction as a regularizer
• Improving Feature-based Visual Localization by Geometry-Aided Matching
• Learning to Localize in Unseen Scenes with Relative Pose Regressors
• 3D reconstruction using Structure for Motion

Cited by in corpus (3)

• Evaluating ML Robustness in GNSS Interference Classification, Characterization & Localization
• PDRNN: Modular Data-driven Pedestrian Dead Reckoning on Loosely Coupled Radio- and Inertial-Signalstreams
• Exploitation of Hidden Context in Dynamic Movement Forecasting: A Neural Network Journey from Recurrent to Graph Neural Networks and General Purpose Transformers