Learning of feature points without additional supervision improves reinforcement learning from images

Abstract

In many control problems that include vision, optimal controls can be inferred from the location of the objects in the scene. This information can be represented using feature points, which is a list of spatial locations in learned feature maps of an input image. Previous works show that feature points learned using unsupervised pre-training or human supervision can provide good features for control tasks. In this paper, we show that it is possible to learn efficient feature point representations end-to-end, without the need for unsupervised pre-training, decoders, or additional losses. Our proposed architecture consists of a differentiable feature point extractor that feeds the coordinates of the estimated feature points directly to a soft actor-critic agent. The proposed algorithm yields performance competitive to the state-of-the art on DeepMind Control Suite tasks.

References in corpus (12)

• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Soft Actor-Critic Algorithms and Applications
• Searching for Activation Functions
• Recurrent World Models Facilitate Policy Evolution
• Learning Latent Dynamics for Planning from Pixels
• Visual Reinforcement Learning with Imagined Goals
• Unsupervised Learning of Object Keypoints for Perception and Control
• Unsupervised Learning of Object Landmarks through Conditional Image Generation
• Loss is its own Reward: Self-Supervision for Reinforcement Learning
• Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning
• End-to-End Robotic Reinforcement Learning without Reward Engineering
• The Ingredients of Real-World Robotic Reinforcement Learning