Reward Design for Driver Repositioning Using Multi-Agent Reinforcement Learning

Abstract

A large portion of passenger requests is reportedly unserviced, partially due to vacant for-hire drivers' cruising behavior during the passenger seeking process. This paper aims to model the multi-driver repositioning task through a mean field multi-agent reinforcement learning (MARL) approach that captures competition among multiple agents. Because the direct application of MARL to the multi-driver system under a given reward mechanism will likely yield a suboptimal equilibrium due to the selfishness of drivers, this study proposes a reward design scheme with which a more desired equilibrium can be reached. To effectively solve the bilevel optimization problem with upper level as the reward design and the lower level as a multi-agent system, a Bayesian optimization (BO) algorithm is adopted to speed up the learning process. We then apply the bilevel optimization model to two case studies, namely, e-hailing driver repositioning under service charge and multiclass taxi driver repositioning under NYC congestion pricing. In the first case study, the model is validated by the agreement between the derived optimal control from BO and that from an analytical solution. With a simple piecewise linear service charge, the objective of the e-hailing platform can be increased by 8.4%. In the second case study, an optimal toll charge of $5.1 is solved using BO, which improves the objective of city planners by 7.9%, compared to that without any toll charge. Under this optimal toll charge, the number of taxis in the NYC central business district is decreased, indicating a better traffic condition, without substantially increasing the crowdedness of the subway system.

arXiv comment28 pages, 20 figures, published in Transportation Research Part C 119 (2020) 102738

References in corpus (16)

• A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning
• Deep Reinforcement Learning for Multi-Agent Systems: A Review of Challenges, Solutions and Applications
• Learning to Communicate with Deep Multi-Agent Reinforcement Learning
• Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
• A Tutorial on Bayesian Optimization
• Mean Field Multi-Agent Reinforcement Learning
• Reinforcement Learning for Solving the Vehicle Routing Problem
• Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms
• Optimal Passenger-Seeking Policies on E-hailing Platforms Using Markov Decision Process and Imitation Learning
• CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms
• Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matching
• Deep Reinforcement Learning Algorithm for Dynamic Pricing of Express Lanes with Multiple Access Locations
• Optimizing Online Matching for Ride-Sourcing Services with Multi-Agent Deep Reinforcement Learning
• A Deep Reinforcement Learning Algorithm Using Dynamic Attention Model for Vehicle Routing Problems
• No-Regret Bayesian Optimization with Unknown Hyperparameters
• Using reinforcement learning to minimize taxi idle times

Cited by in corpus (5)

• Reinforcement Learning for Ridesharing: An Extended Survey
• Multi-Agent Reinforcement Learning for Markov Routing Games: A New Modeling Paradigm For Dynamic Traffic Assignment
• Deep Reinforcement Learning for Demand Driven Services in Logistics and Transportation Systems: A Survey
• SPO-VCS: An End-to-End Smart Predict-then-Optimize Framework with Alternating Differentiation Method for Relocation Problems in Large-Scale Vehicle Crowd Sensing
• DROP: Deep relocating option policy for optimal ride-hailing vehicle repositioning