CLQT: A Closed-Loop, Cost-Aware, Strategy-Consistent Benchmark for Diagnostic Evaluation of LLM Portfolio-Management Agents

Abstract

LLM agents are increasingly cast as autonomous portfolio managers, and benchmarks have moved from financial question-answering to sequential trading. Yet most still rank agents by returns over a fixed window, a weak proxy: the market path dominates a period's return, and apparent alpha can dissolve once look-ahead leakage is controlled. We introduce CLQT, which reframes closed-loop trading evaluation as diagnosis before ranking: an instrument that localizes where and why an agent's process succeeds or fails. CLQT is a closed-loop, cost-aware, strategy-consistent, temporally-gated environment whose agents run a five-stage cycle: gather, synthesize, allocate, execute, reflect. Every round emits a DecisionRound sealed into a recompute-verifiable hash chain. Six pillars form the substrate: a TimeGate, transaction- and financing-cost modeling, strategy-consistency scoring, three-tier memory, a Model-Context-Protocol tool layer, and mandate-aware synthesis. The same agent runs structured or fully autonomous, making process scaffolding an experimental variable. From the audit trail we compute a five-axis capability scorecard (APM-CS: Coherence, Acuity, Composure, Discipline, Reliability), with Coherence partly judged by a held-out, out-of-cohort LLM. We validate CLQT on a contamination-controlled multi-model backtest with an ablation grid and a live broker track on unseen, post-cutoff data, against a repeated-run noise floor. Three findings illustrate the diagnosis-first read: the capability leader is not the Sharpe leader (the nominal winner is a 5-of-26-round reliability artifact); signal-action agreement exceeds a held-out judge's coherence score by a stable margin on both tracks (+0.30 backtest, +0.23 live); and module value registers on the capability axes when returns cannot separate it. CLQT yields not a model ranking but a durable, extensible map of agent competencies and limitations.

56 pages, 15 figures, 15 tables