Learning Logic Specifications for Policy Guidance in POMDPs: an Inductive Logic Programming Approach

Abstract

Partially Observable Markov Decision Processes (POMDPs) are a powerful framework for planning under uncertainty. They allow to model state uncertainty as a belief probability distribution. Approximate solvers based on Monte Carlo sampling show great success to relax the computational demand and perform online planning. However, scaling to complex realistic domains with many actions and long planning horizons is still a major challenge, and a key point to achieve good performance is guiding the action-selection process with domain-dependent policy heuristics which are tailored for the specific application domain. We propose to learn high-quality heuristics from POMDP traces of executions generated by any solver. We convert the belief-action pairs to a logical semantics, and exploit data- and time-efficient Inductive Logic Programming (ILP) to generate interpretable belief-based policy specifications, which are then used as online heuristics. We evaluate thoroughly our methodology on two notoriously challenging POMDP problems, involving large action spaces and long planning horizons, namely, rocksample and pocman. Considering different state-of-the-art online POMDP solvers, including POMCP, DESPOT and AdaOPS, we show that learned heuristics expressed in Answer Set Programming (ASP) yield performance superior to neural networks and similar to optimal handcrafted task-specific heuristics within lower computational time. Moreover, they well generalize to more challenging scenarios not experienced in the training phase (e.g., increasing rocks and grid size in rocksample, incrementing the size of the map and the aggressivity of ghosts in pocman).

References in corpus (19)

• Incremental Pruning: A Simple, Fast, Exact Method for Partially Observable Markov Decision Processes
• DESPOT: Online POMDP Planning with Regularization
• Autonomous task planning and situation awareness in robotic surgery
• Scaling up Heuristic Planning with Relational Decision Trees
• Foundations for Restraining Bolts: Reinforcement Learning with LTLf/LDLf restraining specifications
• Approximate information state for approximate planning and reinforcement learning in partially observed systems
• Deliberation in autonomous robotic surgery: a framework for handling anatomical uncertainty
• Heuristic-Guided Reinforcement Learning
• Logic programming for deliberative robotic task planning
• POMP: Pomcp-based Online Motion Planning for active visual search in indoor environments
• Intention-Aware Navigation in Crowds with Extended-Space POMDP Planning
• Multiagent Rollout and Policy Iteration for POMDP with Application to Multi-Robot Repair Problems
• Axiom Learning and Belief Tracing for Transparent Decision Making in Robotics
• Identification of Unexpected Decisions in Partially Observable Monte-Carlo Planning: a Rule-Based Approach
• Learning Logic Specifications for Soft Policy Guidance in POMCP
• LEADER: Learning Attention over Driving Behaviors for Planning under Uncertainty
• Autonomous tissue retraction with a biomechanically informed logic based framework
• Closing the Planning-Learning Loop with Application to Autonomous Driving
• Inductive Synthesis of Finite-State Controllers for POMDPs