Delay-Aware Reinforcement Learning for O-RAN Control under Stochastic Reward Feedback
arXiv:2611.19344 · doi:10.1109/LNET.2026.3699505
Abstract
Reinforcement learning (RL) is increasingly employed for real-time control in Open Radio Access Networks (O-RANs). However, the asynchronous nature of O-RAN feedback introduces stochastic delays between actions and their observable rewards, leading to misaligned credit assignment and unstable training. This letter formulates the O-RAN control problem as a Markov decision process (MDP) with delayed and stochastic rewards, where the true reward corresponding to an action is received after an unknown delay. To address this, we propose a delay-aware deep RL framework that integrates a long short-term memory (LSTM)-based delay predictor and a temporal reward realignment module within an off-policy learning pipeline. Using empirical delay traces collected from the Open AI Cellular (OAIC) testbed, we demonstrate that the proposed framework achieves faster convergence and significantly reduces latency violations and packet drops compared to conventional RL methods that ignore feedback delay. The results confirm that reward-delay compensation is critical for reliable AI-driven O-RAN control.
References in corpus (5)
- RUDDER: Return Decomposition for Delayed Rewards
- O-RAN: Analysis of Latency-critical Interfaces and Overview of Time Sensitive Networking Solutions
- Off-Policy Reinforcement Learning with Delayed Rewards
- Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks
- Reinforcement Learning with Delayed, Composite, and Partially Anonymous Reward