Deep Reinforcement Learning for Dynamic Multichannel Access in Wireless Networks

Abstract

We consider a dynamic multichannel access problem, where multiple correlated channels follow an unknown joint Markov model. A user at each time slot selects a channel to transmit data and receives a reward based on the success or failure of the transmission. The objective is to find a policy that maximizes the expected long-term reward. The problem is formulated as a partially observable Markov decision process (POMDP) with unknown system dynamics. To overcome the challenges of unknown system dynamics as well as prohibitive computation, we apply the concept of reinforcement learning and implement a Deep Q-Network (DQN) that can deal with large state space without any prior knowledge of the system dynamics. We provide an analytical study on the optimal policy for fixed-pattern channel switching with known system dynamics and show through simulations that DQN can achieve the same optimal performance without knowing the system statistics. We compare the performance of DQN with a Myopic policy and a Whittle Index-based heuristic through both simulations as well as real-data trace and show that DQN achieves near-optimal performance in more complex situations. Finally, we propose an adaptive DQN approach with the capability to adapt its learning in time-varying, dynamic scenarios.

References in corpus (9)

• Adam: A Method for Stochastic Optimization
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Sequence to Sequence Learning with Neural Networks
• Continuous control with deep reinforcement learning
• Playing Atari with Deep Reinforcement Learning
• End-to-End Training of Deep Visuomotor Policies
• Multiple Object Recognition with Visual Attention
• On Myopic Sensing for Multi-Channel Opportunistic Access: Structure, Optimality, and Performance
• Data-Efficient Learning of Feedback Policies from Image Pixels using Deep Dynamical Models