Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning

Abstract

While reinforcement learning has achieved considerable successes in recent years, state-of-the-art models are often still limited by the size of state and action spaces. Model-free reinforcement learning approaches use some form of state representations and the latest work has explored embedding techniques for actions, both with the aim of achieving better generalization and applicability. However, these approaches consider only states or actions, ignoring the interaction between them when generating embedded representations. In this work, we establish the theoretical foundations for the validity of training a reinforcement learning agent using embedded states and actions. We then propose a new approach for jointly learning embeddings for states and actions that combines aspects of model-free and model-based reinforcement learning, which can be applied in both discrete and continuous domains. Specifically, we use a model of the environment to obtain embeddings for states and actions and present a generic architecture that leverages these to learn a policy. In this way, the embedded representations obtained via our approach enable better generalization over both states and actions by capturing similarities in the embedding spaces. Evaluations of our approach on several gaming, robotic control, and recommender systems show it significantly outperforms state-of-the-art models in both discrete/continuous domains with large state/action spaces, thus confirming its efficacy.

References in corpus (16)

• Efficient Estimation of Word Representations in Vector Space
• Playing Atari with Deep Reinforcement Learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Recurrent World Models Facilitate Policy Evolution
• Model-Based Reinforcement Learning for Atari
• Deep Reinforcement Learning in Large Discrete Action Spaces
• A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
• Language Understanding for Text-based Games Using Deep Reinforcement Learning
• Deep Reinforcement Learning based Recommendation with Explicit User-Item Interactions Modeling
• Learning Invariant Representations for Reinforcement Learning without Reconstruction
• Metrics for Finite Markov Decision Processes
• On Learning to Think: Algorithmic Information Theory for Novel Combinations of Reinforcement Learning Controllers and Recurrent Neural World Models
• Dynamics-aware Embeddings
• Learning to Factor Policies and Action-Value Functions: Factored Action Space Representations for Deep Reinforcement learning
• Latent World Models For Intrinsically Motivated Exploration
• Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning

Cited by in corpus (1)

• Low-Dimensional State and Action Representation Learning with MDP Homomorphism Metrics