A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem

Abstract

Financial portfolio management is the process of constant redistribution of a fund into different financial products. This paper presents a financial-model-free Reinforcement Learning framework to provide a deep machine learning solution to the portfolio management problem. The framework consists of the Ensemble of Identical Independent Evaluators (EIIE) topology, a Portfolio-Vector Memory (PVM), an Online Stochastic Batch Learning (OSBL) scheme, and a fully exploiting and explicit reward function. This framework is realized in three instants in this work with a Convolutional Neural Network (CNN), a basic Recurrent Neural Network (RNN), and a Long Short-Term Memory (LSTM). They are, along with a number of recently reviewed or published portfolio-selection strategies, examined in three back-test experiments with a trading period of 30 minutes in a cryptocurrency market. Cryptocurrencies are electronic and decentralized alternatives to government-issued money, with Bitcoin as the best-known example of a cryptocurrency. All three instances of the framework monopolize the top three positions in all experiments, outdistancing other compared trading algorithms. Although with a high commission rate of 0.25% in the backtests, the framework is able to achieve at least 4-fold returns in 50 days.

30 pages, 5 figures, submitting to JMLR

Cited by in corpus (28)

• Deep Graph Convolutional Reinforcement Learning for Financial Portfolio Management -- DeepPocket
• Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization
• FinRL-Podracer: High Performance and Scalable Deep Reinforcement Learning for Quantitative Finance
• Learning Action Representations for Reinforcement Learning
• An Information-Theoretic Optimality Principle for Deep Reinforcement Learning
• Deep Reinforcement Learning for Long-Short Portfolio Optimization
• Trading the Twitter Sentiment with Reinforcement Learning
• Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey
• Jointly-Learned State-Action Embedding for Efficient Reinforcement Learning
• Automatic Financial Trading Agent for Low-risk Portfolio Management using Deep Reinforcement Learning
• PAGAN: Portfolio Analysis with Generative Adversarial Networks
• Reinforcement Learning for Quantitative Trading
• DeepFolio: Convolutional Neural Networks for Portfolios with Limit Order Book Data
• Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution
• Reinforcement-Learning based Portfolio Management with Augmented Asset Movement Prediction States
• Stock2Vec: A Hybrid Deep Learning Framework for Stock Market Prediction with Representation Learning and Temporal Convolutional Network
• A* Tree Search for Portfolio Management
• Deep Reinforcement Learning for Portfolio Optimization using Latent Feature State Space (LFSS) Module
• Deep Online Learning with Stochastic Constraints
• Quantitative Trading using Deep Q Learning
• A General Framework on Enhancing Portfolio Management with Reinforcement Learning
• Deep equal risk pricing of financial derivatives with non-translation invariant risk measures
• Application of Deep Q-Network in Portfolio Management
• A Deep Deterministic Policy Gradient-based Strategy for Stocks Portfolio Management
• Off-Policy Optimization of Portfolio Allocation Policies under Constraints
• Efficient Reinforcement Learning in Resource Allocation Problems Through Permutation Invariant Multi-task Learning
• FinBrain: When Finance Meets AI 2.0
• Nonparametric Online Learning Using Lipschitz Regularized Deep Neural Networks