Distributive Stochastic Learning for Delay-Optimal OFDMA Power and Subband Allocation

Abstract

In this paper, we consider the distributive queue-aware power and subband allocation design for a delay-optimal OFDMA uplink system with one base station, $K$ users and $N_F$ independent subbands. Each mobile has an uplink queue with heterogeneous packet arrivals and delay requirements. We model the problem as an infinite horizon average reward Markov Decision Problem (MDP) where the control actions are functions of the instantaneous Channel State Information (CSI) as well as the joint Queue State Information (QSI). To address the distributive requirement and the issue of exponential memory requirement and computational complexity, we approximate the subband allocation Q-factor by the sum of the per-user subband allocation Q-factor and derive a distributive online stochastic learning algorithm to estimate the per-user Q-factor and the Lagrange multipliers (LM) simultaneously and determine the control actions using an auction mechanism. We show that under the proposed auction mechanism, the distributive online learning converges almost surely (with probability 1). For illustration, we apply the proposed distributive stochastic learning framework to an application example with exponential packet size distribution. We show that the delay-optimal power control has the {\em multi-level water-filling} structure where the CSI determines the instantaneous power allocation and the QSI determines the water-level. The proposed algorithm has linear signaling overhead and computational complexity $\mathcal{O}(KN)$, which is desirable from an implementation perspective.

To appear in Transactions on Signal Processing

Cited by in corpus (6)

• Delay-aware BS Discontinuous Transmission Control and User Scheduling for Energy Harvesting Downlink Coordinated MIMO Systems
• Joint Optimization of File Placement and Delivery in Cache-Assisted Wireless Networks with Limited Lifetime and Cache Space
• Joint Status Sampling and Updating for Minimizing Age of Information in the Internet of Things
• Dynamic Pricing under Finite Space Demand Uncertainty: A Multi-Armed Bandit with Dependent Arms
• Joint Content Delivery and Caching Placement via Dynamic Programming
• Scheduling for Mobile Edge Computing with Random User Arrivals: An Approximate MDP and Reinforcement Learning Approach