End-to-End Egospheric Spatial Memory
arXiv:2102.07764
Abstract
Spatial memory, or the ability to remember and recall specific locations and objects, is central to autonomous agents' ability to carry out tasks in real environments. However, most existing artificial memory modules are not very adept at storing spatial information. We propose a parameter-free module, Egospheric Spatial Memory (ESM), which encodes the memory in an ego-sphere around the agent, enabling expressive 3D representations. ESM can be trained end-to-end via either imitation or reinforcement learning, and improves both training efficiency and final performance against other memory baselines on both drone and manipulator visuomotor control tasks. The explicit egocentric geometry also enables us to seamlessly combine the learned controller with other non-learned modalities, such as local obstacle avoidance. We further show applications to semantic segmentation on the ScanNet dataset, where ESM naturally combines image-level and map-level inference modalities. Through our broad set of experiments, we show that ESM provides a general computation graph for embodied spatial reasoning, and the module forms a bridge between real-time mapping systems and differentiable memory architectures. Implementation at: https://github.com/ivy-dl/memory.
Conference paper at ICLR 2021. Implementation: https://github.com/ivy-dl/memory Project page: https://djl11.github.io/ESM/
References in corpus (8)
- End-To-End Memory Networks
- IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
- Unsupervised Predictive Memory in a Goal-Directed Agent
- Transferring End-to-End Visuomotor Control from Simulation to Real World for a Multi-Stage Task
- Neural Map: Structured Memory for Deep Reinforcement Learning
- Sim-to-Real Reinforcement Learning for Deformable Object Manipulation
- PyRep: Bringing V-REP to Deep Robot Learning
- Ivy: Templated Deep Learning for Inter-Framework Portability