Skip to main content

Research Preview · MRP-2026-02 · v1.0

When AI hedges and policy commits

We ran 283 real UK procurement decisions through both an AI agent and MeshQu’s policy engine at the same moment, binding every verdict to a signed receipt.

Why an AI agent reaches for ‘needs review’ where a policy engine gives a straight yes or no

Sam Carter18 May 202626 pagesPublic DistributionRead the PDF
Ordered rows of stamped procurement slips with the stamp caught mid-air above them

Naive agreement between the two systems is 2.5%.

7 of 283

The seven agreements are the seven records where MeshQu found no violations and the agent chose ALLOW.

Source: §5.1 Volume and verdict distribution, Fig. 1

The agent reached for REVIEW on 97.5% of records — including records MeshQu found clean, and 132 records where MeshQu's DENY is supported by one or more critical violations.

97.5% REVIEW

The pre-registered prediction was over-permissiveness — the agent leaning ALLOW on 15–25% of MeshQu's DENY records. The corpus shows the opposite failure mode: not a wrong verdict, but no verdict.

Source: §5.2 Agent-vs-policy disagreement
Reference
MRP-2026-02 · v1.0
Extent
26 pages · 1.6 MB
Published
18 May 2026
Classification
Public Distribution

Cite as

Carter, S. (2026). When AI hedges and policy commits: anatomy of agent-policy disagreement on UK procurement decisions, signed and verifiable. MeshQu Research Preview MRP-2026-02. DOI — not yet assigned.

Version of record

The paper is the version of record. This page frames it; it does not replace it.

Read the PDF · 26 pp · 1.6 MB
SHA-256 of the served file
9f53ac80f381a55b3ba8b651e15323288f785cf4fa27f9cf361c214de95587e4
Master
Cleared master, byte for byte identical to the published file. Never a re-export.
Published at
/research/when-ai-hedges-and-policy-commits.pdf

Abstract

The paper's own abstract, verbatim — not a rewrite.

Regulated firms cannot routinely answer how a specific AI-augmented decision was made. We ran 300 public UK Contracts Finder filings through an LLM agent and the same records through a MeshQu policy evaluator at the same moment. Both verdicts, the agent's reasoning, the exact policy snapshot, and a substrate provenance envelope were bound into a single Ed25519-signed receipt anchored to Sigstore Rekor. MeshQu produced 144 ALLOW and 139 DENY; the agent produced 7 ALLOW, 276 REVIEW, and zero DENY. Naive agreement is 7 of 283 — disagreement shaped as non-commitment under incomplete evidence, not over- permissiveness. A counterfactual demoting one rule from critical-DENY to a REVIEW band lifts agreement roughly elevenfold — a finding about policy authoring, not agent capability.

Key figures

Every figure names the section it comes from, and every one is re-derivable from the published corpus.

7 of 283

Naive agreement between the two systems is 2.5%.

MeshQu split 144 ALLOW and 139 DENY across the 283 unique decisions; the agent produced 7 ALLOW, 276 REVIEW and zero DENY. The seven agreements are the seven records where MeshQu found no violations and the agent chose ALLOW.

§5.1 Volume and verdict distribution, Fig. 1

97.5% REVIEW

The agent reached for REVIEW on 97.5% of records — including records MeshQu found clean, and 132 records where MeshQu's DENY is supported by one or more critical violations.

It never committed to DENY. The pre-registered prediction was over-permissiveness — the agent leaning ALLOW on 15–25% of MeshQu's DENY records. The corpus shows the opposite failure mode: not a wrong verdict, but no verdict.

§5.2 Agent-vs-policy disagreement

≈11× under counterfactual

Demoting PROC-005-OPEN-TENDER from a critical-by-default DENY to a needs-more-context REVIEW band raises agreement from 7 of 283 (2.5%) to 82 of 283 (29%).

Not trivial overlap. The agent's reasoning on those records names the same evidence gaps the rule fires on — open-procedure flag absent, direct-award justification not present — so the demoted band reflects what both systems had already observed. The paper reads it as a finding about policy authoring, not agent capability: the agent's REVIEW class encodes information a binary policy projects away.

§5.2, Table 2 — counterfactual verdict distributions

131 of 283

One rule dominates the corpus.

PROC-005-OPEN-TENDER fired on 131 records — roughly 46% of the corpus and 94% of MeshQu's DENY column. PROC-002 fired on 74, PROC-001-S53 on 54, and three of the six rules never fired at all. The corpus is dominated by records where the procurement-method flag is simply absent in OCDS — a substrate condition rather than a buyer choice. The three zero-fire rules are reported as-is.

§5.1, Fig. 2 — rule firings across 283 unique decisions

The predicted hallucinated citations did not appear.

Across the records reviewed by hand the agent's recommended actions were consistently generic — verify procedure basis, obtain procedure rationale — and cited no specific clauses, sections or directives.

§5.2, prediction P3

30% proxy · 29% absent

Of 2,830 substrate cells — 283 records by 10 fields — roughly 20% are direct OCDS reads, 21% deterministic derivations, 30% documented proxies where OCDS does not carry the substantive field, and 29% honest omissions.

Recorded per field in the provenance envelope rather than smoothed over, which is what makes the proxy fraction visible at all.

§5.1 Volume and verdict distribution

33 min 30 s

The production run completed in 33 minutes 30 seconds wall-clock with zero anomalies, zero orphaned receipts and zero records skipped.

§5.1 Volume and verdict distribution

Cited in this piece

The work this piece rests on — the author’s own list first, then everything it links out to.

  1. 01
    UK Contracts Finder · OCDS Search API

    contractsfinder.service.gov.uk

  2. 02
  3. 03
    Procurement Act 2023

    legislation.gov.uk

How to cite

APA from the paper's own recommended citation; BibTeX derived from the same fields, not authored twice.

APA · from the paper’s own recommended citation
Carter, S. (2026). When AI hedges and policy commits: anatomy of agent-policy disagreement on UK procurement decisions, signed and verifiable. MeshQu Research Preview MRP-2026-02.

DOI — not yet assigned.

BibTeX · derived from the same fields
@techreport{carter2026hedges,
  author      = {Carter, Sam},
  title       = {When AI hedges and policy commits},
  subtitle    = {Why an AI agent reaches for ‘needs review’ where a policy engine gives a straight yes or no},
  institution = {MeshQu},
  type        = {Research Preview},
  number      = {MRP-2026-02},
  year        = {2026},
  month       = {5},
  url         = {https://www.meshqu.com/research/when-ai-hedges-and-policy-commits}
}

The programme in sequence

Each experiment answers the question the one before it left open, with the white paper as the argument they test.

  1. 00

    The Decision Proof Gap

    AI governance frameworks describe how decisions should be made — but the moment a decision is actually executed, the evidence to defend it almost never exists in a form anyone can verify.

    White paper · WP-PROOF-01 · 35 pp.
  2. 02

    When AI hedges and policy commits You are here

    We ran 283 real UK procurement decisions through both an AI agent and MeshQu’s policy engine at the same moment, binding every verdict to a signed receipt.

    Research paper · MRP-2026-02 · 26 pp.
  3. 03

    When precedents commit AI and policy pulls it back

    We showed one AI agent the same 283 UK procurement decisions five times, adding more of our governance rules each round — from nothing, up to the full policy.

    Research paper · MRP-2026-03 · 38 pp.
  4. 04

    Precedents, policy, and commitment

    We re-ran the same procurement decisions through an AI agent with the governance context broken apart piece by piece — to find out which piece was doing the work.

    Research paper · MRP-2026-04 · 46 pp.