Model-based Deep Reinforcement Learning for Dynamic Portfolio Optimization

Abstract

Dynamic portfolio optimization is the process of sequentially allocating wealth to a collection of assets in some consecutive trading periods, based on investors' return-risk profile. Automating this process with machine learning remains a challenging problem. Here, we design a deep reinforcement learning (RL) architecture with an autonomous trading agent such that, investment decisions and actions are made periodically, based on a global objective, with autonomy. In particular, without relying on a purely model-free RL agent, we train our trading agent using a novel RL architecture consisting of an infused prediction module (IPM), a generative adversarial data augmentation module (DAM) and a behavior cloning module (BCM). Our model-based approach works with both on-policy or off-policy RL algorithms. We further design the back-testing and execution engine which interact with the RL agent in real time. Using historical {\em real} financial market data, we simulate trading with practical constraints, and demonstrate that our proposed model is robust, profitable and risk-sensitive, as compared to baseline trading strategies and model-free RL agents from prior work.

References in corpus (12)

• Continuous control with deep reinforcement learning
• Trust Region Policy Optimization
• Prioritized Experience Replay
• Conditional Time Series Forecasting with Convolutional Neural Networks
• Parameter Space Noise for Exploration
• Towards Principled Methods for Training Generative Adversarial Networks
• Real-valued (Medical) Time Series Generation with Recurrent Conditional GANs
• Memory-based control with recurrent neural networks
• A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem
• Adversarial Deep Reinforcement Learning in Portfolio Management
• Time-series modeling with undecimated fully convolutional neural networks
• Robust Log-Optimal Strategy with Reinforcement Learning

Cited by in corpus (9)

• The Evolution of Reinforcement Learning in Quantitative Finance: A Survey
• Multi-Agent Deep Reinforcement Learning for Liquidation Strategy Analysis
• Asynchronous Deep Double Duelling Q-Learning for Trading-Signal Execution in Limit Order Book Markets
• Model-Free Reinforcement Learning for Financial Portfolios: A Brief Survey
• Decision-Focused Fine-Tuning of Time Series Foundation Models for Dispatchable Feeder Optimization
• Reinforcement Learning for Quantitative Trading
• Deep Stock Trading: A Hierarchical Reinforcement Learning Framework for Portfolio Optimization and Order Execution
• Deep Reinforcement Learning for Portfolio Optimization using Latent Feature State Space (LFSS) Module
• Off-Policy Optimization of Portfolio Allocation Policies under Constraints