Conditional Driving from Natural Language Instructions
arXiv:1910.07615
Abstract
Widespread adoption of self-driving cars will depend not only on their safety but largely on their ability to interact with human users. Just like human drivers, self-driving cars will be expected to understand and safely follow natural-language directions that suddenly alter the pre-planned route according to user's preference or in presence of ambiguities, particularly in locations with poor or outdated map coverage. To this end, we propose a language-grounded driving agent implementing a hierarchical policy using recurrent layers and gated attention. The hierarchical approach enables us to reason both in terms of high-level language instructions describing long time horizons and low-level, complex, continuous state/action spaces required for real-time control of a self-driving car. We train our policy with conditional imitation learning from realistic language data collected from human drivers and navigators. Through quantitative and interactive experiments within the CARLA framework, we show that our model can successfully interpret language instructions and follow them safely, even when generalizing to previously unseen environments. Code and video are available at https://sites.google.com/view/language-grounded-driving.
Accepted by the 3rd Conference on Robot Learning, Osaka, Japan (CoRL 2019)
References in corpus (12)
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Continuous control with deep reinforcement learning
- Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation
- End to End Learning for Self-Driving Cars
- Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
- MobileNetV2: Inverted Residuals and Linear Bottlenecks
- Modular Multitask Reinforcement Learning with Policy Sketches
- Self-Monitoring Navigation Agent via Auxiliary Progress Estimation
- FollowNet: Robot Navigation by Following Natural Language Directions with Deep Reinforcement Learning
- TACO: Learning Task Decomposition via Temporal Alignment for Control
- Tactical Rewind: Self-Correction via Backtracking in Vision-and-Language Navigation
- DDCO: Discovery of Deep Continuous Options for Robot Learning from Demonstrations