Distributed Uplink Beamforming in Cell-Free Networks Using Deep Reinforcement Learning

Abstract

The emergence of new wireless technologies together with the requirement of massive connectivity results in several technical issues such as excessive interference, high computational demand for signal processing, and lengthy processing delays. In this work, we propose several beamforming techniques for an uplink cell-free network with centralized, semi-distributed, and fully distributed processing, all based on deep reinforcement learning (DRL). First, we propose a fully centralized beamforming method that uses the deep deterministic policy gradient algorithm (DDPG) with continuous space. We then enhance this method by enabling distributed experience at access points (AP). Indeed, we develop a beamforming scheme that uses the distributed distributional deterministic policy gradients algorithm (D4PG) with the APs representing the distributed agents. Finally, to decrease the computational complexity, we propose a fully distributed beamforming scheme that divides the beamforming computations among APs. The results show that the D4PG scheme with distributed experience achieves the best performance irrespective of the network size. Furthermore, the proposed distributed beamforming technique performs better than the DDPG algorithm with centralized learning only for small-scale networks. The performance superiority of the DDPG model becomes more evident as the number of APs and/or users increases. Moreover, during the operation stage, all DRL models demonstrate a significantly shorter processing time than that of the conventional gradient descent (GD) solution.

References in corpus (10)

• Continuous control with deep reinforcement learning
• A Coordinated Approach to Channel Estimation in Large-scale Multiple-antenna Systems
• Scalable Cell-Free Massive MIMO Systems
• DeepMind Control Suite
• Distributed Distributional Deterministic Policy Gradients
• Local Partial Zero-Forcing Precoding for Cell-Free Massive MIMO
• Scalability Aspects of Cell-Free Massive MIMO
• Centralized and Distributed Power Allocation for Max-Min Fairness in Cell-Free Massive MIMO
• Max-Min Optimal Beamforming for Cell-Free Massive MIMO
• Multiple Access in Dynamic Cell-Free Networks: Outage Performance and Deep Reinforcement Learning-Based Design