From Language Models to Control Policies: Search, Selection, and Verification

Abstract

Industrial process control demands policies that are interpretable and auditable, requirements that black-box neural policies struggle to meet. LLM-driven heuristic synthesis offers an alternative: a language model iteratively proposes and refines human-readable Python controllers using rich behavioral feedback from a simulator. We study this form of programmatic policy search on hot steel rolling, a physics-simulated, multi-pass control problem with competing objectives and hard equipment constraints, and address three questions that arise once the search is run at scale: how to split a fixed budget of LLM iterations between restarting and refining, how to deploy the resulting pool of heuristics, and how to verify it. Treating each search run as a randomized algorithm, we find across 52 independent runs that runs of moderate length match the hindsight-optimal budget allocation, and that Luby-style universal restarts reach this regime without problem-specific tuning. Because each heuristic is cheap to simulate, a small portfolio selected per job outperforms the best single heuristic, also under cross-validation and on out-of-distribution operating conditions on which the best heuristic stalls. Finally, an automated audit pipeline combining static analysis, Z3 proofs, and property-based testing proves key safety properties for all 174 synthesized heuristics; the stall, which neither the search reward nor these properties reveal, shows that the specifications must also cover progress.