Horizon: Facebook's Open Source Applied Reinforcement Learning Platform

Abstract

In this paper we present Horizon, Facebook's open source applied reinforcement learning (RL) platform. Horizon is an end-to-end platform designed to solve industry applied RL problems where datasets are large (millions to billions of observations), the feedback loop is slow (vs. a simulator), and experiments must be done with care because they don't run in a simulator. Unlike other RL platforms, which are often designed for fast prototyping and experimentation, Horizon is designed with production use cases as top of mind. The platform contains workflows to train popular deep RL algorithms and includes data preprocessing, feature transformation, distributed training, counterfactual policy evaluation, optimized serving, and a model-based data understanding tool. We also showcase and describe real examples where reinforcement learning models trained with Horizon significantly outperformed and replaced supervised learning systems at Facebook.

arXiv comment10 pages

References in corpus (10)

• Continuous control with deep reinforcement learning
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Trust Region Policy Optimization
• Dueling Network Architectures for Deep Reinforcement Learning
• Noisy Networks for Exploration
• Deep Reinforcement Learning for Page-wise Recommendations
• Recommendations with Negative Feedback via Pairwise Deep Reinforcement Learning
• Doubly Robust Policy Evaluation and Learning
• Device Placement Optimization with Reinforcement Learning
• Making Contextual Decisions with Low Technical Debt

Cited by in corpus (34)

• A Minimalist Approach to Offline Reinforcement Learning
• Benchmarking Batch Deep Reinforcement Learning Algorithms
• Harms from Increasingly Agentic Algorithmic Systems
• Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
• rlpyt: A Research Code Base for Deep Reinforcement Learning in PyTorch
• RecSim: A Configurable Simulation Platform for Recommender Systems
• An empirical investigation of the challenges of real-world reinforcement learning
• Reinforcement Learning with Combinatorial Actions: An Application to Vehicle Routing
• Reinforcement Learning Applications
• Reinforcement Learning for Slate-based Recommender Systems: A Tractable Decomposition and Practical Methodology
• Knowledge Transfer for Cross-Domain Reinforcement Learning: A Systematic Review
• Rule Mining over Knowledge Graphs via Reinforcement Learning
• A View on Deep Reinforcement Learning in System Optimization
• CAQL: Continuous Action Q-Learning
• Temporal Multimodal Multivariate Learning
• An Equivalence between Loss Functions and Non-Uniform Sampling in Experience Replay
• RecSim NG: Toward Principled Uncertainty Modeling for Recommender Ecosystems
• SLM Lab: A Comprehensive Benchmark and Modular Software Framework for Reproducible Deep Reinforcement Learning
• Lessons from Contextual Bandit Learning in a Customer Support Bot
• Catalyst.RL: A Distributed Framework for Reproducible RL Research
• Fiber: A Platform for Efficient Development and Distributed Training for Reinforcement Learning and Population-Based Methods
• Distilled Thompson Sampling: Practical and Efficient Thompson Sampling via Imitation Learning
• Privacy-Constrained Policies via Mutual Information Regularized Policy Gradients
• Human and Multi-Agent collaboration in a human-MARL teaming framework
• Baconian: A Unified Open-source Framework for Model-Based Reinforcement Learning
• Sample Efficient Ensemble Learning with Catalyst.RL
• Beyond Our Behavior: The GDPR and Humanistic Personalization
• The AI Arena: A Framework for Distributed Multi-Agent Reinforcement Learning
• Bellman: A Toolbox for Model-Based Reinforcement Learning in TensorFlow
• Discovering an Aid Policy to Minimize Student Evasion Using Offline Reinforcement Learning
• Techniques Toward Optimizing Viewability in RTB Ad Campaigns Using Reinforcement Learning
• Simultaneous Relevance and Diversity: A New Recommendation Inference Approach
• Showing Your Offline Reinforcement Learning Work: Online Evaluation Budget Matters
• Efficient Reinforcement Learning Development with RLzoo