RAFT: Recurrent All-Pairs Field Transforms for Optical Flow

Abstract

We introduce Recurrent All-Pairs Field Transforms (RAFT), a new deep network architecture for optical flow. RAFT extracts per-pixel features, builds multi-scale 4D correlation volumes for all pairs of pixels, and iteratively updates a flow field through a recurrent unit that performs lookups on the correlation volumes. RAFT achieves state-of-the-art performance. On KITTI, RAFT achieves an F1-all error of 5.10%, a 16% error reduction from the best published result (6.10%). On Sintel (final pass), RAFT obtains an end-point-error of 2.855 pixels, a 30% error reduction from the best published result (4.098 pixels). In addition, RAFT has strong cross-dataset generalization as well as high efficiency in inference time, training speed, and parameter count. Code is available at https://github.com/princeton-vl/RAFT.

arXiv commentfixed a formatting issue, Eq 7. no change in content

References in corpus (8)

• Decoupled Weight Decay Regularization
• Deep Equilibrium Models
• DeepV2D: Video to Depth with Differentiable Structure from Motion
• Differentiable Convex Optimization Layers
• BA-Net: Dense Bundle Adjustment Network
• Trellis Networks for Sequence Modeling
• A Lightweight Optical Flow CNN - Revisiting Data Fidelity and Regularization
• Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation

Cited by in corpus (15)

• Advances in Deep Concealed Scene Understanding
• Dynamic Facial Expression Recognition under Partial Occlusion with Optical Flow Reconstruction
• Can Temporal Information Help with Contrastive Self-Supervised Learning?
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses
• Asphalt Concrete Characterization Using Digital Image Correlation: A Systematic Review of Best Practices, Applications, and Future Vision
• Efficient Regional Memory Network for Video Object Segmentation
• Automatic Generation of Dense Non-rigid Optical Flow
• Progressive Temporal Feature Alignment Network for Video Inpainting
• End-to-End Entity Detection with Proposer and Regressor
• Generalizing to the Open World: Deep Visual Odometry with Online Adaptation
• Let's See Clearly: Contaminant Artifact Removal for Moving Cameras
• Tangent Space Backpropagation for 3D Transformation Groups
• On the Sensory Commutativity of Action Sequences for Embodied Agents
• Deep Two-View Structure-from-Motion Revisited
• Merging Tasks for Video Panoptic Segmentation