# When AI hedges and policy commits

Canonical: https://www.meshqu.com/research/when-ai-hedges-and-policy-commits

We ran 283 real UK procurement decisions through both an AI agent and MeshQu’s policy engine at the same moment, binding every verdict to a signed receipt.

Published: 2026-05-18T00:00:00.000Z

## Key facts

- 7 of 283 — Naive agreement between the two systems is 2.5%. MeshQu split 144 ALLOW and 139 DENY across the 283 unique decisions; the agent produced 7 ALLOW, 276 REVIEW and zero DENY. (The seven agreements are the seven records where MeshQu found no violations and the agent chose ALLOW.)
- 97.5% REVIEW — The agent reached for REVIEW on 97.5% of records — including records MeshQu found clean, and 132 records where MeshQu's DENY is supported by one or more critical violations. It never committed to DENY. (The pre-registered prediction was over-permissiveness — the agent leaning ALLOW on 15–25% of MeshQu's DENY records. The corpus shows the opposite failure mode: not a wrong verdict, but no verdict.)
- ≈11× under counterfactual — Demoting PROC-005-OPEN-TENDER from a critical-by-default DENY to a needs-more-context REVIEW band raises agreement from 7 of 283 (2.5%) to 82 of 283 (29%). (Not trivial overlap. The agent's reasoning on those records names the same evidence gaps the rule fires on — open-procedure flag absent, direct-award justification not present — so the demoted band reflects what both systems had already observed. The paper reads it as a finding about policy authoring, not agent capability: the agent's REVIEW class encodes information a binary policy projects away.)
- 131 of 283 — One rule dominates the corpus. PROC-005-OPEN-TENDER fired on 131 records — roughly 46% of the corpus and 94% of MeshQu's DENY column. PROC-002 fired on 74, PROC-001-S53 on 54, and three of the six rules never fired at all. (The corpus is dominated by records where the procurement-method flag is simply absent in OCDS — a substrate condition rather than a buyer choice. The three zero-fire rules are reported as-is.)
- The predicted hallucinated citations did not appear. Across the records reviewed by hand the agent's recommended actions were consistently generic — verify procedure basis, obtain procedure rationale — and cited no specific clauses, sections or directives.
- 30% proxy · 29% absent — Of 2,830 substrate cells — 283 records by 10 fields — roughly 20% are direct OCDS reads, 21% deterministic derivations, 30% documented proxies where OCDS does not carry the substantive field, and 29% honest omissions. (Recorded per field in the provenance envelope rather than smoothed over, which is what makes the proxy fraction visible at all.)
- 33 min 30 s — The production run completed in 33 minutes 30 seconds wall-clock with zero anomalies, zero orphaned receipts and zero records skipped.

{/* The PDF remains the version of record. The summary, method, findings
    and limitations above are frontmatter, not body prose — they render
    through the research template's structured blocks (PSA-040) so that
    each one is separately labelled, quotable and checkable against the
    section of the paper it came from. The MDX body stays empty by
    design: this page frames and summarises the publication, it does not
    become it. */}