Visuomotor Understanding for Representation Learning of Driving Scenes
arXiv:1909.06979
Abstract
Dashboard cameras capture a tremendous amount of driving scene video each day. These videos are purposefully coupled with vehicle sensing data, such as from the speedometer and inertial sensors, providing an additional sensing modality for free. In this work, we leverage the large-scale unlabeled yet naturally paired data for visual representation learning in the driving scenario. A representation is learned in an end-to-end self-supervised framework for predicting dense optical flow from a single frame with paired sensing data. We postulate that success on this task requires the network to learn semantic and geometric knowledge in the ego-centric view. For example, forecasting a future view to be seen from a moving vehicle requires an understanding of scene depth, scale, and movement of objects. We demonstrate that our learned representation can benefit other tasks that require detailed scene understanding and outperforms competing unsupervised representations on semantic segmentation.
arXiv commentBMVC 2019. Supplementary material: https://bmvc2019.org/wp-content/uploads/papers/0002-supplementary.zip Dataset: http://github.com/SeokjuLee/driving-dataset-doc
References in corpus (11)
- Very Deep Convolutional Networks for Large-Scale Image Recognition
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
- The Cityscapes Dataset for Semantic Urban Scene Understanding
- Deep Convolutional Inverse Graphics Network
- Weakly-supervised Disentangling with Recurrent Transformations for 3D View Synthesis
- Unsupervised Learning of Depth and Ego-Motion from Video
- Learning a Driving Simulator
- GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose
- Mid-Level Visual Representations Improve Generalization and Sample Efficiency for Learning Visuomotor Policies
- Transformation-Grounded Image Generation Network for Novel 3D View Synthesis
- Geometry-aware Deep Network for Single-Image Novel View Synthesis