Deep reinforcement learning for optimal trading with partial information

Abstract

Reinforcement Learning (RL) has attracted increasing interest in financial applications, including optimal trading and execution. However, the use of RL for optimal trading strategies that exploit latent information in the market has been, to the best of our knowledge, subject to little attention. In this paper, we consider an optimal trading problem in which the trading signal follows an Ornstein-Uhlenbeck process with regime switching parameters. The problem is naturally formulated as a partially observable Markov decision problem, requiring a trader to infer latent information directly from the history of the observable process. We combine recurrent neural networks (RNN) with RL to address both the filtering and trading components of the problem. More specifically, we propose three distinct RL-based approaches, on top of which we incorporate RNN to filter the latent state of the environment where the agent is trading. The first, a one-step approach, directly encodes hidden states from the GRU into the RL trader. The second and third are two-step methods: one that uses posterior regime probability estimates for the mean reverting regimes, while the other relies on forecasts of the next signal value. Through extensive simulations with increasingly complex Markovian regime dynamics for the trading signal's parameters, as well as an empirical application to equity pairs trading, we find that feeding posterior probability estimates on the latent long-run mean reversion regime achieves superior cumulative rewards and exhibits more interpretable strategies, in contrast with the other two approaches. Overall, our results show that the quality and structure of the information supplied to the agent are important for trading performance and policy interpretability.