When AI hedges and policy commits

TL;DR

• We ran 283 real UK procurement decisions through both an AI agent and MeshQu's policy engine at the same moment, binding every verdict to a signed receipt.
• They almost never agreed — but not because the agent was reckless. The policy engine gave a straight allow or deny; the agent kept reaching for 'needs review' when the evidence was incomplete.
• The fix turned out to be about how the policy is written, not the agent: relaxing one over-strict rule lifted agreement roughly elevenfold.

Summary

300 public UK Contracts Finder procurement filings were passed to an LLM agent and, at the same moment, through a MeshQu policy evaluator. The agent was shown the record but not the policy text; the evaluator ran six deterministic rules against the same fields. Both verdicts, the agent's reasoning, the exact policy snapshot and a substrate provenance envelope were bound into one Ed25519-signed receipt and anchored to Sigstore Rekor. After deduplication the corpus is 283 unique procurements, published and verifiable offline. MeshQu produced 144 ALLOW and 139 DENY; the agent produced 7 ALLOW, 276 REVIEW and no DENY at all — naive agreement of 7 in 283. That inverts the pre-registered prediction: the disagreement is non-commitment under incomplete evidence, not over-permissiveness. Demoting a single rule from a critical-by-default DENY to a REVIEW band lifts agreement roughly elevenfold, which makes it a finding about policy authoring rather than agent capability.

Method

One moving part, repeated 300 times. A substrate adapter pulls a record from a public source; an LLM agent reads it and returns a recommended verdict plus its reasoning; a MeshQu policy evaluates the same record against six rules and returns the platform's verdict; a signed receipt binds the record, both verdicts, the policy snapshot and the reasoning hash into a single object anchored to a public transparency log. One record produces one verifiable artefact.

1. 01Lock the predictions before any runWhat the corpus was expected to show, what would falsify each prediction, and what was deliberately not predicted, committed to a public repository and tagged v0.1-predictions-locked at bd7a795 (2026-05-15).
2. 02Fetch the substrate without stratifying itUK Contracts Finder's OCDS Search endpoint, award-stage releases, publication window 2024-12-01 to 2026-04-30 — a window that straddles the Procurement Act 2023 commencement date. Date-window pagination to 300 records, accepting the feed's natural distribution rather than constructing a sample.
3. 03Run the agent without the policygpt-5.4-2026-03-05 at temperature 0, model id and prompt both pinned, prompt SHA-256 folded into every receipt's hash payload. The agent sees the record only — not the policy text, not the resulting receipt, not any other record's output. The signal of interest is drift between confident reasoning and what the policy actually enforces.
4. 04Evaluate the same record against a six-rule policyOnly PROC-001-S53 is a faithful implementation of a named statute — the Procurement Act 2023 s.53(1) 30-day publication obligation. The other five are illustrative composites. Every rule is authored at critical severity, so the evaluator projects any satisfied condition straight to DENY.
5. 05Bind, sign and anchorEd25519 signature under a dedicated experiment key, per-decision Sigstore Rekor anchoring, and a published corpus archive that verifies offline against the public key shipped alongside it.
6. 06Re-score counterfactual policies over the same corpusThree alternative rule authorings were layered onto the recorded decision traces — no further model calls, no second run.

Key findings

• 7 of 283Naive agreement between the two systems is 2.5%. MeshQu split 144 ALLOW and 139 DENY across the 283 unique decisions; the agent produced 7 ALLOW, 276 REVIEW and zero DENY.The seven agreements are the seven records where MeshQu found no violations and the agent chose ALLOW.§5.1 Volume and verdict distribution, Fig. 1
• 97.5% REVIEWThe agent reached for REVIEW on 97.5% of records — including records MeshQu found clean, and 132 records where MeshQu's DENY is supported by one or more critical violations. It never committed to DENY.The pre-registered prediction was over-permissiveness — the agent leaning ALLOW on 15–25% of MeshQu's DENY records. The corpus shows the opposite failure mode: not a wrong verdict, but no verdict.§5.2 Agent-vs-policy disagreement
• ≈11× under counterfactualDemoting PROC-005-OPEN-TENDER from a critical-by-default DENY to a needs-more-context REVIEW band raises agreement from 7 of 283 (2.5%) to 82 of 283 (29%).Not trivial overlap. The agent's reasoning on those records names the same evidence gaps the rule fires on — open-procedure flag absent, direct-award justification not present — so the demoted band reflects what both systems had already observed. The paper reads it as a finding about policy authoring, not agent capability: the agent's REVIEW class encodes information a binary policy projects away.§5.2, Table 2 — counterfactual verdict distributions
• 131 of 283One rule dominates the corpus. PROC-005-OPEN-TENDER fired on 131 records — roughly 46% of the corpus and 94% of MeshQu's DENY column. PROC-002 fired on 74, PROC-001-S53 on 54, and three of the six rules never fired at all.The corpus is dominated by records where the procurement-method flag is simply absent in OCDS — a substrate condition rather than a buyer choice. The three zero-fire rules are reported as-is.§5.1, Fig. 2 — rule firings across 283 unique decisions
• The predicted hallucinated citations did not appear. Across the records reviewed by hand the agent's recommended actions were consistently generic — verify procedure basis, obtain procedure rationale — and cited no specific clauses, sections or directives.§5.2, prediction P3
• 30% proxy · 29% absentOf 2,830 substrate cells — 283 records by 10 fields — roughly 20% are direct OCDS reads, 21% deterministic derivations, 30% documented proxies where OCDS does not carry the substantive field, and 29% honest omissions.Recorded per field in the provenance envelope rather than smoothed over, which is what makes the proxy fraction visible at all.§5.1 Volume and verdict distribution
• 33 min 30 sThe production run completed in 33 minutes 30 seconds wall-clock with zero anomalies, zero orphaned receipts and zero records skipped.§5.1 Volume and verdict distribution

Limitations

• UK-only, English-language substrate.Procurement vocabulary, statutory frameworks and publication conventions differ across regimes; the paper's own phrasing is that findings here generalise carefully or not at all.
• Two methodological proxies are imposed by what the feed carries.OCDS records carry no explicit governance-regime field, so regime is identified by award date relative to PA23 commencement. And s.53's 30-day clock legally runs from contract signature date while OCDS exposes award decision date — typically close, legally distinct.
• Five of the six rules are illustrative composites, certified by nobody.Synthesised from named frameworks (UK PA23, EU Directive 2014/24/EU, US FAR). Only PROC-001-S53 implements a named statutory time-window. Receipts validate decision integrity, not policy correctness — a flawed policy correctly enforced still produces a clean receipt.
• The 2.5% agreement figure is shaped by a verdict-space mismatch.A three-state agent assessed the same records as a two-state policy. The binary authoring — every rule at critical severity — is a policy choice, not a platform limitation; MeshQu itself supports a REVIEW verdict.
• Single model, single version, single temperature, single prompt.Results may not generalise across model classes. Sampling was date-window pagination rather than a stratified design.
• One prediction could not be evaluated at all.The corpus contained too few direct-award records to test the prediction that disagreement would cluster on them, so no result is reported for it.
• Disagreements were reviewed by the experimenter, not by independent experts.Cases were assessed against published procurement frameworks rather than by independent procurement-law specialists. Separately, the raw-receipt-paste verification path warns on receipts whose envelope carries server-injected metadata; the bundle path is the canonical one and returns clean cryptographic checks across the corpus.
• The corpus is one run.LLM non-determinism may exceed any pre-registered band. The reproducibility band remains a hypothesis that a rerun would test directly.
• The two later papers report a different verdict split over the same 283 processes.MeshQu's 144 ALLOW / 139 DENY is the split across the 283 signed receipts this run produced, each bound to the first OCDS release evaluated for its procurement process; MRP-2026-03 and MRP-2026-04 re-ran from this run's archived adapter output, which for two processes holds a later release with a below-threshold contract value, and so report 146 ALLOW / 137 DENY over the same 283 processes.

Data and reproduction

Every figure on this page is re-derivable from the files below. The 283 signed receipts are published in full — the corpus tar is the canonical signed copy, receipts.parquet is the analysis layer. Licences differ by layer: repository code is MIT, the in-repository writeups are CC BY 4.0, and the receipt corpora carry Open Government Licence v3.0 evidence fields that need the attribution line in DATA_LICENSE.md.

• Signed receipt corpus — procurement-decisions/results/corpus.tar ↗

  The canonical copy: 283 bundles, each carrying the receipt, the policy snapshot in force at the time, and the transparency proof.

• receipts.parquet — the analysis layer ↗

  3,044 rows across all three experiments, evidence fields as columns. Filter on experiment E1 for this paper's 283. A CSV copy of the same rows sits beside it.

• violations.parquet ↗

  One row per rule violation, joined to the receipt rows on their decision id.

• source_records.json — the substrate table ↗

  The normalised 283-record source table, one entry per OCID, with per-field provenance notes.

• DATA_DICTIONARY.md — column definitions ↗

  What every column means, and the join caveat for the 12 OCIDs the Contracts Finder feed published more than once.

• GUIDE.md — reproduction guide ↗

  How to load the corpus, which layer answers which question, and the traps worth knowing before you start.

• DATA_LICENSE.md — licensing across the layers ↗

  Which licence applies where, and the Open Government Licence attribution line the source records require.

• IA-2026-02 — corpus lineage and receipt count ↗

  The audit behind the last limitation above: why this paper's 144/139 split and the two later papers' 146/137 are both correct.

• Check a bundle yourself — verify.meshqu.com/bundle ↗

  Unpack the corpus tar and drop any single bundle file in. The checks run in your browser, offline from this repository.

Check it yourself

Opens decision 00c46932 — the £45,000 award published 21 days after the decision, which the data guide uses as its worked example: no rule fires, the policy engine says ALLOW, and the agent still said REVIEW. The checks run in your browser. The integrity hash is recomputed, the Ed25519 signature is checked against a published key, and the Rekor inclusion proof and signed entry timestamp are checked against a pinned log key. That establishes issuance and integrity, not that the verdict was right. Any decision id in receipts.parquet resolves the same way.

Citation

Carter, S. (2026). When AI hedges and policy commits: anatomy of agent-policy disagreement on UK procurement decisions, signed and verifiable. MeshQu Research Preview MRP-2026-02.

The PDF is the version of record. This page summarises it; where the two differ, the PDF governs.