Event-based Photometric Bundle Adjustment

Abstract

We tackle the problem of bundle adjustment (i.e., simultaneous refinement of camera poses and scene map) for a purely rotating event camera. Starting from first principles, we formulate the problem as a classical non-linear least squares optimization. The photometric error is defined using the event generation model directly in the camera rotations and the semi-dense scene brightness that triggers the events. We leverage the sparsity of event data to design a tractable Levenberg-Marquardt solver that handles the very large number of variables involved. To the best of our knowledge, our method, which we call Event-based Photometric Bundle Adjustment (EPBA), is the first event-only photometric bundle adjustment method that works on the brightness map directly and exploits the space-time characteristics of event data, without having to convert events into image-like representations. Comprehensive experiments on both synthetic and real-world datasets demonstrate EPBA's effectiveness in decreasing the photometric error (by up to 90%), yielding results of unparalleled quality. The refined maps reveal details that were hidden using prior state-of-the-art rotation-only estimation methods. The experiments on modern high-resolution event cameras show the applicability of EPBA to panoramic imaging in various scenarios (without map initialization, at multiple resolutions, and in combination with other methods, such as IMU dead reckoning or previous event-based rotation estimation methods). We make the source code publicly available. https://github.com/tub-rip/epba

21 pages, 19 figures, 10 tables. Project page: https://github.com/tub-rip/epba

References in corpus (15)

• Past, Present, and Future of Simultaneous Localization And Mapping: Towards the Robust-Perception Age
• Event-based Vision: A Survey
• The Event-Camera Dataset and Simulator: Event-based Data for Pose Estimation, Visual Odometry, and SLAM
• Ultimate SLAM? Combining Events, Images, and IMU for Robust Visual SLAM in HDR and High Speed Scenarios
• A Unifying Contrast Maximization Framework for Event Cameras, with Applications to Motion, Depth, and Optical Flow Estimation
• Event-aided Direct Sparse Odometry
• A compact formula for the derivative of a 3-D rotation in exponential coordinates
• Efficient Derivative Computation for Cumulative B-Splines on Lie Groups
• CMax-SLAM: Event-based Rotational-Motion Bundle Adjustment and SLAM System using Contrast Maximization
• Globally-Optimal Contrast Maximisation for Event Cameras
• The Spatio-Temporal Poisson Point Process: A Simple Model for the Alignment of Event Camera Data
• Formulating Event-based Image Reconstruction as a Linear Inverse Problem with Deep Regularization using Optical Flow
• Event-based Stereo Visual Odometry with Native Temporal Resolution via Continuous-time Gaussian Process Regression
• Motion-prior Contrast Maximization for Dense Continuous-Time Motion Estimation
• Event-based Mosaicing Bundle Adjustment