Optimal and Scalable Caching for 5G Using Reinforcement Learning of Space-time Popularities
arXiv:1708.06698 · doi:10.1109/JSTSP.2017.2787979
Abstract
Small basestations (SBs) equipped with caching units have potential to handle the unprecedented demand growth in heterogeneous networks. Through low-rate, backhaul connections with the backbone, SBs can prefetch popular files during off-peak traffic hours, and service them to the edge at peak periods. To intelligently prefetch, each SB must learn what and when to cache, while taking into account SB memory limitations, the massive number of available contents, the unknown popularity profiles, as well as the space-time popularity dynamics of user file requests. In this work, local and global Markov processes model user requests, and a reinforcement learning (RL) framework is put forth for finding the optimal caching policy when the transition probabilities involved are unknown. Joint consideration of global and local popularity demands along with cache-refreshing costs allow for a simple, yet practical asynchronous caching approach. The novel RL-based caching relies on a Q-learning algorithm to implement the optimal policy in an online fashion, thus enabling the cache control unit at the SB to learn, track, and possibly adapt to the underlying dynamics. To endow the algorithm with scalability, a linear function approximation of the proposed Q-learning scheme is introduced, offering faster convergence as well as reduced complexity and memory requirements. Numerical tests corroborate the merits of the proposed approach in various realistic settings.
References in corpus (2)
Cited by in corpus (28)
- In-Edge AI: Intelligentizing Mobile Edge Computing, Caching and Communication by Federated Learning
- Applying Machine Learning Techniques for Caching in Edge Networks: A Comprehensive Survey
- Proximal Policy Optimization-based Transmit Beamforming and Phase-shift Design in an IRS-aided ISAC System for the THz Band
- Distributed Reinforcement Learning for Privacy-Preserving Dynamic Edge Caching
- Towards Understanding Asynchronous Advantage Actor-critic: Convergence and Linear Speedup
- Unsupervised Recurrent Federated Learning for Edge Popularity Prediction in Privacy-Preserving Mobile Edge Computing Networks
- A View on Deep Reinforcement Learning in System Optimization
- An Overview of Analysis Methods and Evaluation Results for Caching Strategies
- Online Caching with Optimistic Learning
- Optimistic No-regret Algorithms for Discrete Caching
- Using Grouped Linear Prediction and Accelerated Reinforcement Learning for Online Content Caching
- Online Reinforcement Learning of X-Haul Content Delivery Mode in Fog Radio Access Networks
- Collaborative Multi-Agent Multi-Armed Bandit Learning for Small-Cell Caching
- Caching Transient Content for IoT Sensing: Multi-Agent Soft Actor-Critic
- Dynamic Coded Caching in Wireless Networks Using Multi-Agent Reinforcement Learning
- Learning and Management for Internet-of-Things: Accounting for Adaptivity and Scalability
- A rolling-horizon dynamic programming approach for collaborative caching
- Learning Automata Based Q-learning for Content Placement in Cooperative Caching
- Joint Long-Term Cache Updating and Short-Term Content Delivery in Cloud-Based Small Cell Networks
- Multi-Agent Reinforcement Learning for Cooperative Coded Caching via Homotopy Optimization
- Learning to Cache With No Regrets
- Distributed Edge Caching via Reinforcement Learning in Fog Radio Access Networks
- Hybrid Policy Learning for Energy-Latency Tradeoff in MEC-Assisted VR Video Service
- Optimal Caching Designs for Perfect, Imperfect and Unknown File Popularity Distributions in Large-Scale Multi-Tier Wireless Networks
- Robust, Deep, and Reinforcement Learning for Management of Communication and Power Networks
- Distributed Network Caching via Dynamic Programming
- Dynamic Content Updates in Heterogeneous Wireless Networks
- Reinforcement Learning Based Cooperative Coded Caching under Dynamic Popularities in Ultra-Dense Networks