Model-Free Learning of Optimal Ergodic Policies in Wireless Systems

Abstract

Learning optimal resource allocation policies in wireless systems can be effectively achieved by formulating finite dimensional constrained programs which depend on system configuration, as well as the adopted learning parameterization. The interest here is in cases where system models are unavailable, prompting methods that probe the wireless system with candidate policies, and then use observed performance to determine better policies. This generic procedure is difficult because of the need to cull accurate gradient estimates out of these limited system queries. This paper constructs and exploits smoothed surrogates of constrained ergodic resource allocation problems, the gradients of the former being representable exactly as averages of finite differences that can be obtained through limited system probing. Leveraging this unique property, we develop a new model-free primal-dual algorithm for learning optimal ergodic resource allocations, while we rigorously analyze the relationships between original policy search problems and their surrogates, in both primal and dual domains. First, we show that both primal and dual domain surrogates are uniformly consistent approximations of their corresponding original finite dimensional counterparts. Upon further assuming the use of near-universal policy parameterizations, we also develop explicit bounds on the gap between optimal values of initial, infinite dimensional resource allocation problems, and dual values of their parameterized smoothed surrogates. In fact, we show that this duality gap decreases at a linear rate relative to smoothing and universality parameters. Thus, it can be made arbitrarily small at will, also justifying our proposed primal-dual algorithmic recipe. Numerical simulations confirm the effectiveness of our approach.

arXiv comment13 pages, 4 figures

References in corpus (10)

• Learning to Optimize: Training Deep Neural Networks for Wireless Resource Management
• Multi-Agent Deep Reinforcement Learning for Dynamic Power Allocation in Wireless Networks
• Optimal Wireless Resource Allocation with Random Edge Graph Neural Networks
• Spatial Deep Learning for Wireless Scheduling
• Learning Optimal Resource Allocations in Wireless Systems
• Random Access Design for Wireless Control Systems
• Asynchronous Incremental Stochastic Dual Descent Algorithm for Network Resource Allocation
• Learning in Wireless Control Systems over Non-Stationary Channels
• Zeroth-Order Algorithms for Nonconvex Minimax Problems with Improved Complexities
• Energy-Efficient Subchannel and Power Allocation for HetNets Based on Convolutional Neural Network

Cited by in corpus (5)

• A Tutorial on Ultra-Reliable and Low-Latency Communications in 6G: Integrating Domain Knowledge into Deep Learning
• Zeroth-order Deterministic Policy Gradient
• Robust and Reliable Stochastic Resource Allocation via Tail Waterfilling
• Asynchronous Message-Passing and Zeroth-Order Optimization Based Distributed Learning with a Use-Case in Resource Allocation in Communication Networks
• A Deep Learning Based Resource Allocator for Communication Networks with Dynamic User Utility Demands