Less is More: Consistent Video Depth Estimation with Masked Frames Modeling
arXiv:2208.00380 · doi:10.1145/3503161.3547978
Abstract
Temporal consistency is the key challenge of video depth estimation. Previous works are based on additional optical flow or camera poses, which is time-consuming. By contrast, we derive consistency with less information. Since videos inherently exist with heavy temporal redundancy, a missing frame could be recovered from neighboring ones. Inspired by this, we propose the frame masking network (FMNet), a spatial-temporal transformer network predicting the depth of masked frames based on their neighboring frames. By reconstructing masked temporal features, the FMNet can learn intrinsic inter-frame correlations, which leads to consistency. Compared with prior arts, experimental results demonstrate that our approach achieves comparable spatial accuracy and higher temporal consistency without any additional information. Our work provides a new perspective on consistent video depth estimation. Our official project page is https://github.com/RaymondWang987/FMNet.
arXiv commentAccepted by ACM MM 2022
References in corpus (9)
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
- Distilling the Knowledge in a Neural Network
- Deformable DETR: Deformable Transformers for End-to-End Object Detection
- BEiT: BERT Pre-Training of Image Transformers
- DepthTransfer: Depth Extraction from Video Using Non-parametric Sampling
- From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation
- DeepV2D: Video to Depth with Differentiable Structure from Motion
- ConvTransformer: A Convolutional Transformer Network for Video Frame Synthesis
- Unsupervised Learning of Monocular Depth Estimation with Bundle Adjustment, Super-Resolution and Clip Loss