PlaTe: Visually-Grounded Planning with Transformers in Procedural Tasks
arXiv:2109.04869 · doi:10.1109/LRA.2022.3150855
Abstract
In this work, we study the problem of how to leverage instructional videos to facilitate the understanding of human decision-making processes, focusing on training a model with the ability to plan a goal-directed procedure from real-world videos. Learning structured and plannable state and action spaces directly from unstructured videos is the key technical challenge of our task. There are two problems: first, the appearance gap between the training and validation datasets could be large for unstructured videos; second, these gaps lead to decision errors that compound over the steps. We address these limitations with Planning Transformer (PlaTe), which has the advantage of circumventing the compounding prediction errors that occur with single-step models during long model-based rollouts. Our method simultaneously learns the latent state and action information of assigned tasks and the representations of the decision-making process from human demonstrations. Experiments conducted on real-world instructional videos and an interactive environment show that our method can achieve a better performance in reaching the indicated goal than previous algorithms. We also validated the possibility of applying procedural tasks on a UR-5 platform. We make our code publicly available and support academic research purposes.
References in corpus (10)
- Adam: A Method for Stochastic Optimization
- Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling
- Learning Latent Dynamics for Planning from Pixels
- AI2-THOR: An Interactive 3D Environment for Visual AI
- Towards Automatic Learning of Procedures from Web Instructional Videos
- Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updates
- Dynamics Learning with Cascaded Variational Inference for Multi-Step Manipulation
- Egocentric Human Trajectory Forecasting with a Wearable Camera and Multi-Modal Fusion
- Modular Action Concept Grounding in Semantic Video Prediction
- Zero-Shot Anticipation for Instructional Activities