MASPRM: Multi-Agent System Process Reward Model
arXiv:2510.24803
The paper introduces MASPRM, a model that scores ordered message transcripts in multi‑agent systems to guide inference‑time search methods such as step‑level beam search and Monte Carlo Tree Search, using only terminal outcome rewards for training.
Abstract
Inference-time search over multi-agent systems (MAS) wastes compute when it cannot identify which agent's intermediate message advanced progress. We present the Multi-Agent System Process Reward Model (MASPRM), which scores routed transcripts (ordered sequences of messages between agents) and acts as an inference controller for step-level beam search (SBS) and Monte Carlo Tree Search (MCTS). MASPRM is trained from multi-agent MCTS rollouts labeled only with terminal outcome rewards, without human step-level annotations. We evaluate on GSM8K, MATH, MMLU, and LogiQA. Under matched scorer size and comparable MCTS budget, MASPRM exceeds a size-matched ORM by to points at 1.5B and to at 7B across all four benchmarks, with additional scorer-scaling gains over policy likelihood at 7B (avg under MCTS). MASPRM also improves ranking quality, reducing Hit@1 to Hit@5 gaps by up to points, with the largest gains under stepwise search that uses intermediate decisions. Code: https://github.com/milad1378yz/MASPRM