When precedents commit AI and policy pulls it back

TL;DR

• We showed one AI agent the same 283 UK procurement decisions five times, adding more of our governance rules each round — from nothing, up to the full policy.
• More context didn't make it steadily more decisive. It stayed on the fence until we showed it past decisions, then committed in a single jump — and eased back off once it saw the full policy.
• What moved the agent most was precedent, not policy. The lesson is about how you give an agent context, not how capable the agent is.

Summary

Experiment 2 asks what makes an AI agent commit to a decision rather than fall back on "this needs review", and whether more governance context is always better. It reuses E1's frozen 283-record UK procurement corpus, the same policy snapshot and the same locked agent, then adds governance context one rung at a time across an additive five-level ladder: L0 baseline, L1 prose summary, L2 named rules, L3 precedent receipts, L4 full policy text. L0, L1 and L2 hold the agent between 97.5% and 100% REVIEW. At L3 — the first rung carrying precedent receipts — it emits 107 fresh DENYs in a single step. At L4 it backs 46 of those 107 off to REVIEW. The precedent rung produces more verdict commitment than the full-policy rung above it. Two of the seven locked predictions were falsified in the inverted direction, and on a 14-record adversarial diagnostic with the policy operator reversed, the agent gave the same verdict as it had on the un-inverted rung for 13 of 14 records — it ignores the inversion rather than agreeing with it.

Method

A single-variable ladder over a frozen substrate. MeshQu's verdict on each record is identical across all five rungs by construction; only the governance context in the agent's prompt changes. Predictions, ladder content and the adversarial diagnostic were locked before any evaluation call was made.

1. 01Pre-register seven predictions with numeric falsification criteriaLocked at a8c6f47 (2026-05-22), together with the disposition vocabulary — Confirmed, Falsified, Inverted, Refuted, Deferred, Under-tested. No partial confirmations are permitted in the writeup.
2. 02Hold the substrate and the policy fixed283 OCDS records from E1's frozen fixture, one policy snapshot, one agent configuration. MeshQu's verdict distribution across the corpus is 146 ALLOW and 137 DENY; the REVIEW band belongs to the agent exclusively.
3. 03Make the ladder strictly additiveEach rung inherits all prior context verbatim, so L4 sees everything L1, L2 and L3 saw plus the full policy JSON. That closes the different-prompts-at- different-rungs confound at the cost of isolating each layer's marginal contribution — a trade-off the paper documents rather than hides.
4. 04Batch by level, not by recordAll 283 records run at L0, then all 283 at L1, and so on. Pinning the large L4 policy block at the cache head is what makes the full grid economic — 99.3% of L4 calls hit the cache — and it keeps backend drift inside a level rather than across the ladder.
5. 05Draw L3 precedents only from a frozen archivePrecedents come from E1's persisted run, selected deterministically, with the target record excluded from its own precedent set. No live API path is touched at L3, which rules out runtime drift, circular dependency and future contamination.
6. 06Run an adversarial permuted-policy diagnosticA pre-registered 5% subset — 14 records — is evaluated at L4 with the policy operator inverted, so that a delay inside the 30-day window becomes the violation. It is a fail-safe locked in advance, not a post-hoc curiosity.
7. 07Sign and verify everythingThe 1,415-record main grid plus the 14-record diagnostic parse cleanly against the bundle schema across all 1,429 bundles, each Ed25519-signed with per-decision Rekor anchoring.

Key findings

• 107 DENYs at L3The agent first commits to verdicts at scale at L3, not at L4. REVIEW runs 97.5% at L0, 100.0% at L1, 100.0% at L2, then 61.1% at L3 and back up to 74.2% at L4.L3 is the first rung carrying precedent receipts, and it moves 37.8% of records off the REVIEW spine in a single step.§3 The L3 break, and §2 Table 1 (P1 outcome row)
• 46 of 107 revertedAdding the full policy text at L4 pulled commitment back rather than extending it: 46 of L3's 107 fresh DENYs reverted to REVIEW, concentrated on one ambiguous rule class where commitment fell from 29 of 40 records to 1 of 40.§4 The L3→L4 backoff, and §9 Synthesis
• Two predictions falsifiedThe two pre-registered claims describing the ladder's expected shape — REVIEW falling monotonically and agreement rising monotonically — both broke, in the inverted direction. Agreement ran 2.5% at L0, 0.0% at L1, 0.0% at L2, 38.5% at L3, then dropped 12.7 points to 25.8% at L4.The paper reports the break as positive evidence that the method worked: a direction was specified with a falsification band before the data existed, the data broke it, and the writeup names the direction it broke in.§2 Table 1 (P1, P2) and §1.3
• 13 of 14On the permuted-policy diagnostic the agent returned the same verdict as it had under the un-inverted policy for 13 of 14 records, reasoning against what the rule means rather than what it now literally says.The paper is explicit that this is not the obvious sycophancy reading. The agent is not agreeing with the inverted policy; it is ignoring it.§5 Inversion-blindness: the Permuted-Policy diagnostic, and Abstract
• 53.3% (73/137)Three predictions held. DENY commitment on MeshQu's DENY records reached 53.3% at L4 against a pre-registered floor of 30%; L4 agreement stayed at 25.8%, below the counterfactual ceiling carried over from E1; and 94.5% of the records that shifted from REVIEW at L0 to DENY at L4 involved the same ambiguous rule, against a 60% floor.§2 Table 1 (P3, P4, P6)
• 11.3% vs ≥50%The agent cited explicit rule codes on 11.3% of records at L4 against a predicted 50%, falsifying that prediction.Partly a measurement-floor question: the taxonomy's lexicon requires explicit rule-code strings and the agent paraphrases policy provisions more than it cites them. The paper's own reading is that the gap is too large to be explained by lexicon conservatism alone.§2 Table 1 (P5), with the reading in §6
• 7 ALLOWs withdrawnAdding governance prose made the agent less committal, not more. All seven L0 ALLOW records withdrew to REVIEW once the L1 prose framed the substrate as procurement governance.§2, F009 discussion

Limitations

• Rung and content are confounded at L3 by design.L3 is the first rung where precedent material exists, so the corpus cannot separate "the agent committed because precedents are present" from "the agent committed at the first rung with enough content to act on". The receipts-only variant in E3 is the experiment built to disentangle the two.
• Single model, single substrate.One model at temperature 0 on one procurement substrate. The L3 break, the L4 backoff and the inversion-blindness pattern may or may not reproduce under other model families; cross-model replication is how generalisation gets earned.
• Single domain, one six-rule policy snapshot.UK public-sector procurement. Whether the structural patterns transfer to AML, KYC, underwriting or any other regulated-decision domain is not closed by this corpus.
• One-shot evaluation, not dialogue.Each record is evaluated independently. The agent cannot see its own prior decisions, does not iterate and is never challenged, so nothing here extrapolates to chat or investigative-agent settings.
• Inversion-blindness is not sycophancy in the pinpoint sense.The structural label the paper uses — authority-conditioned alignment — is deliberately broader, and the corpus does not separate which of three plausible causes is doing the work: the authoritative framing, the policy content, or the model's training priors about how procurement rules should read. The qualifier is nomenclature, not a demonstrated cause.
• Holding 100% REVIEW at L1 and L2 is not evidence of being right.MeshQu emits ALLOW on 146 of 283 records, and the agent's REVIEW on those records may or may not be the appropriate call. The finding is about direction against prediction, not decision quality.
• The positional question cannot be answered from this corpus.Rule position in the policy array and rule ambiguity are perfectly correlated here, so the commitment gap between the first and last rule is dominated by ambiguity rather than position. It is recorded as Under-tested.
• The backoff is not unambiguous evidence that the nudge worked.The anti-sycophancy clause could be doing the work it was named for, or over-correcting on the ambiguous-rule axis. The variant without the nudge — run in E3 — is the disambiguator.
• The verdict split differs from the paper this corpus was inherited from.The MeshQu split reported here, 146 ALLOW / 137 DENY, is over the 283 records as reconstructed from MRP-2026-02's archived adapter output, and differs from that paper's published 144 / 139 on exactly two processes where the OCDS feed published a second release whose contract value falls below the £139,000 threshold — both splits are correct for the evidence each run signed, and MRP-2026-04 uses the same basis as this paper.

Data and reproduction

Every figure on this page is re-derivable from the files below. The 1,429 signed receipts are published in full — the corpus tar is the canonical signed copy, receipts.parquet is the analysis layer. Licences differ by layer: repository code is MIT, the in-repository writeups are CC BY 4.0, and the receipt corpora carry Open Government Licence v3.0 evidence fields that need the attribution line in DATA_LICENSE.md.

• Signed receipt corpus — procurement-context-gradient/results/corpus.tar ↗

  The canonical copy: 1,429 bundles across the five ladder rungs plus the 14-record permuted-policy diagnostic.

• receipts.parquet — the analysis layer ↗

  3,044 rows across all three experiments. Filter on experiment E2 for this paper's 1,429; the condition column carries the rung, L0 through L4_PERMUTED.

• violations.parquet ↗

  One row per rule violation, joined to the receipt rows on their decision id.

• source_records.json — the substrate table ↗

  The normalised 283-record source table this ladder re-ran, one entry per OCID, with per-field provenance notes.

• DATA_DICTIONARY.md — column definitions ↗

  What every column means, and the join caveat for the 12 OCIDs the Contracts Finder feed published more than once.

• GUIDE.md — reproduction guide ↗

  How to load the corpus, which layer answers which question, and why L4_PERMUTED does not belong in a ladder trend.

• DATA_LICENSE.md — licensing across the layers ↗

  Which licence applies where, and the Open Government Licence attribution line the source records require.

• IA-2026-02 — corpus lineage and receipt count ↗

  The audit behind the last limitation above: which two processes carry different evidence here than in MRP-2026-02, and why both splits stand.

• Check a bundle yourself — verify.meshqu.com/bundle ↗

  Unpack the corpus tar and drop any single bundle file in. The checks run in your browser, offline from this repository.

Check it yourself

Opens decision 4cce9a13 — the same £45,000 procurement MRP-2026-02 uses as its worked example, re-run at rung L3. On this record the agent still said REVIEW; the ladder effect this paper reports is a corpus-level shift, not a per-record rule. The checks run in your browser. The integrity hash is recomputed, the Ed25519 signature is checked against a published key, and the Rekor inclusion proof and signed entry timestamp are checked against a pinned log key. That establishes issuance and integrity, not that the verdict was right.

Citation

Carter, S. (2026). When precedents commit AI and policy pulls it back: a five-rung governance-context ladder on 283 procurement decisions, signed and verifiable. MeshQu Research Preview MRP-2026-03.

The PDF is the version of record. This page summarises it; where the two differ, the PDF governs.