Distributive Dynamic Spectrum Access through Deep Reinforcement Learning: A Reservoir Computing Based Approach

Abstract

Dynamic spectrum access (DSA) is regarded as an effective and efficient technology to share radio spectrum among different networks. As a secondary user (SU), a DSA device will face two critical problems: avoiding causing harmful interference to primary users (PUs), and conducting effective interference coordination with other secondary users. These two problems become even more challenging for a distributed DSA network where there is no centralized controllers for SUs. In this paper, we investigate communication strategies of a distributive DSA network under the presence of spectrum sensing errors. To be specific, we apply the powerful machine learning tool, deep reinforcement learning (DRL), for SUs to learn "appropriate" spectrum access strategies in a distributed fashion assuming NO knowledge of the underlying system statistics. Furthermore, a special type of recurrent neural network (RNN), called the reservoir computing (RC), is utilized to realize DRL by taking advantage of the underlying temporal correlation of the DSA network. Using the introduced machine learning-based strategy, SUs could make spectrum access decisions distributedly relying only on their own current and past spectrum sensing outcomes. Through extensive experiments, our results suggest that the RC-based spectrum access strategy can help the SU to significantly reduce the chances of collision with PUs and other SUs. We also show that our scheme outperforms the myopic method which assumes the knowledge of system statistics, and converges faster than the Q-learning method when the number of channels is large.

arXiv commentThis work is accepted in IEEE IoT Journal 2018

References in corpus (3)

• Playing Atari with Deep Reinforcement Learning
• On the difficulty of training Recurrent Neural Networks
• Listen-and-Talk: Full-duplex Cognitive Radio Networks

Cited by in corpus (18)

• Recent Advances in Physical Reservoir Computing: A Review
• Deep-Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks
• Revolutionizing Future Connectivity: A Contemporary Survey on AI-empowered Satellite-based Non-Terrestrial Networks in 6G
• Cognitive Radar Using Reinforcement Learning in Automotive Applications
• Decentralized Automotive Radar Spectrum Allocation to Avoid Mutual Interference Using Reinforcement Learning
• A Deep Reinforcement Learning Framework for Contention-Based Spectrum Sharing
• The RFML Ecosystem: A Look at the Unique Challenges of Applying Deep Learning to Radio Frequency Applications
• Artificial Intelligence-Enabled Cellular Networks: A Critical Path to Beyond-5G and 6G
• Self-Tuning Sectorization: Deep Reinforcement Learning Meets Broadcast Beam Optimization
• Deep Echo State Q-Network (DEQN) and Its Application in Dynamic Spectrum Sharing for 5G and Beyond
• Multi-Agent Deep Reinforcement Learning Multiple Access for Heterogeneous Wireless Networks with Imperfect Channels
• Deep Reinforcement Learning Based Massive Access Management for Ultra-Reliable Low-Latency Communications
• Carrier-Sense Multiple Access for Heterogeneous Wireless Networks Using Deep Reinforcement Learning
• HpGAN: Sequence Search with Generative Adversarial Networks
• Dynamic Multichannel Access via Multi-agent Reinforcement Learning: Throughput and Fairness Guarantees
• Multi-agent reinforcement learning using echo-state network and its application to pedestrian dynamics
• Deep Reinforcement Learning for Dynamic Spectrum Sensing and Aggregation in Multi-Channel Wireless Networks
• Non-Uniform Time-Step Deep Q-Network for Carrier-Sense Multiple Access in Heterogeneous Wireless Networks