The StreetLearn Environment and Dataset

Abstract

Navigation is a rich and well-grounded problem domain that drives progress in many different areas of research: perception, planning, memory, exploration, and optimisation in particular. Historically these challenges have been separately considered and solutions built that rely on stationary datasets - for example, recorded trajectories through an environment. These datasets cannot be used for decision-making and reinforcement learning, however, and in general the perspective of navigation as an interactive learning task, where the actions and behaviours of a learning agent are learned simultaneously with the perception and planning, is relatively unsupported. Thus, existing navigation benchmarks generally rely on static datasets (Geiger et al., 2013; Kendall et al., 2015) or simulators (Beattie et al., 2016; Shah et al., 2018). To support and validate research in end-to-end navigation, we present StreetLearn: an interactive, first-person, partially-observed visual environment that uses Google Street View for its photographic content and broad coverage, and give performance baselines for a challenging goal-driven navigation task. The environment code, baseline agent code, and the dataset are available at http://streetlearn.cc

13 pages, 6 figures, 4 tables. arXiv admin note: text overlap with arXiv:1804.00168

References in corpus (12)

• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes
• Learning to Navigate in Complex Environments
• Matterport3D: Learning from RGB-D Data in Indoor Environments
• AI2-THOR: An Interactive 3D Environment for Visual AI
• Learning to Execute
• CARLA: An Open Urban Driving Simulator
• MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments
• Playing FPS Games with Deep Reinforcement Learning
• HoME: a Household Multimodal Environment
• CHALET: Cornell House Agent Learning Environment
• The AdobeIndoorNav Dataset: Towards Deep Reinforcement Learning based Real-world Indoor Robot Visual Navigation

Cited by in corpus (13)

• A Survey on Approximate Edge AI for Energy Efficient Autonomous Driving Services
• Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directions
• Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View
• Driving Datasets Literature Review
• Towards mental time travel: a hierarchical memory for reinforcement learning agents
• Deep Single Image Camera Calibration by Heatmap Regression to Recover Fisheye Images Under Manhattan World Assumption
• Navigation Agents for the Visually Impaired: A Sidewalk Simulator and Experiments
• You Are Here: Geolocation by Embedding Maps and Images
• Cross-View Policy Learning for Street Navigation
• SILG: The Multi-environment Symbolic Interactive Language Grounding Benchmark
• PanGEA: The Panoramic Graph Environment Annotation Toolkit
• Extreme Rotation Estimation using Dense Correlation Volumes
• Rapid Task-Solving in Novel Environments