Multi-Timescale Ensemble Q-learning for Markov Decision Process Policy Optimization

Abstract

Reinforcement learning (RL) is a classical tool to solve network control or policy optimization problems in unknown environments. The original Q-learning suffers from performance and complexity challenges across very large networks. Herein, a novel model-free ensemble reinforcement learning algorithm which adapts the classical Q-learning is proposed to handle these challenges for networks which admit Markov decision process (MDP) models. Multiple Q-learning algorithms are run on multiple, distinct, synthetically created and structurally related Markovian environments in parallel; the outputs are fused using an adaptive weighting mechanism based on the Jensen-Shannon divergence (JSD) to obtain an approximately optimal policy with low complexity. The theoretical justification of the algorithm, including the convergence of key statistics and Q-functions are provided. Numerical results across several network models show that the proposed algorithm can achieve up to 55% less average policy error with up to 50% less runtime complexity than the state-of-the-art Q-learning algorithms. Numerical results validate assumptions made in the theoretical analysis.

References in corpus (4)

• Measuring and testing dependence by correlation of distances
• $QD$-Learning: A Collaborative Distributed Strategy for Multi-Agent Reinforcement Learning Through Consensus + Innovations
• DQLEL: Deep Q-Learning for Energy-Optimized LoS/NLoS UWB Node Selection
• Closed-Form Delay-Optimal Power Control for Energy Harvesting Wireless System with Finite Energy Storage

Cited by in corpus (1)

• Leveraging Digital Cousins for Ensemble Q-Learning in Large-Scale Wireless Networks