Mapping Instructions and Visual Observations to Actions with Reinforcement Learning

Dipendra Misra, John Langford, Yoav Artzi

We propose to directly map raw visual observations and text input to actions for instruction execution. While existing approaches assume access to structured environment representa…