Why we stayed in shadow mode

Last week we described connecting MeshQu to our own agentic coding workflow. At chosen points, it checks the information our coordinating system supplies against approved policy and records the answer. It advises. It does not stop anything. Every change remains subject to human-approved authority, whether through an individual sign-off or a standing rule agreed in advance. MeshQu's assessments are advisory; they do not authorise merges.

We said we would stay in shadow mode while we tuned the rules. We did. What we did not plan for was how much of the workflow itself we would change in the same week.

Changing the work and the rules at the same time

The agents moved fast this week. Speed has a cost, so we built cost controls mid-week: lighter coordination, leaner agent sessions, and fewer restarts.

We also changed how we work with the agents. On one busy day, roughly a third of the instructions we gave them were friction rather than decisions. We turned the most common ones into standing rules, so the agents proceed and report back instead of waiting to ask. Those rules shape how the agents work. Authorising a change to the product stays with a person.

Each of those changes alters what the controls see. A budget rule behaves differently when sessions get lighter. A merge rule behaves differently when the work arrives in a different shape. If we had let the controls start influencing the work at the same time, we could not have told which change caused what.

Shadow mode kept one thing out of the experiment: MeshQu's assessments did not decide which work could go ahead. It was not a controlled test of the workflow, because the workflow itself changed. But whatever we saw, our own controls had not caused it.

Can the record be re-run?

Before relying on the record to judge anything, we tested the record itself.

On 20 October we replayed all 753 decisions recorded so far, each under the policy version it was made under. Every one came back with the same verdict and the same rule results.

That shows the assessments are reproducible. If a policy changes in July, someone will later ask which version applied to a decision in June, and what it said. A record that replays exactly can answer that. One that only stores a final answer cannot.

Replay has limits. It shows MeshQu would assess the same information the same way. It does not show that the information supplied was accurate, or that the original decision was right.

The same record has a second, different use: testing past cases against a different policy. That is what the next section does.

A revised policy, fewer potential reviews

The most useful result was about our merge policy: the rule that decides whether a change can join the product.

Under the first version, none of 120 recorded merge decisions could have gone ahead without a person looking. We refined it, then replayed the same 120 decisions under version 2. Some changes were recorded more than once, so the 120 decisions cover fewer than 120 changes.

These are replays, not live decisions. In practice, a person authorised every one of those merges. The question was how many decisions version 2 would have let through without one. Same decisions, different policy. And for each one, we can see exactly which rule put it where it landed. The 7 alerts are decisions the policy would have refused; in shadow mode, a refusal is recorded as an alert. They are the same under both versions.

This is the question every control owner faces: if I change this rule, how much checking does it remove, and what does it let through? Here the answer came from our own history, not a guess.

Last week we warned ourselves about the obvious trap. Fewer reviews might mean a better process. It might also mean rules that are easier to pass. 69 decisions cleared on paper, covering 49 changes, is not 69 decisions that were right. We come back to how we are checking that below.

The record caught us

Some of what the record showed was about us, not the agents.

We broke our own budget rule. We have a rule against starting new builder agents once usage passes a set line. This week it fired 34 times. Each time, we had started a builder anyway. Shadow mode meant nothing stopped us, but the record said plainly: you set this limit, and here is each time you went past it.

We were not recording everything we thought we were. Last week we said a receipt cannot show a checkpoint that was never sent. This week that stopped being theoretical. By reconciling the record against the merges that actually happened, we found that 13 of 19 merges on 19 and 20 October had never reached MeshQu. Most of the gaps were caused by a labelling bug. MeshQu could not have spotted those on its own. The reconciliation did. We fixed the bug, and the reconciliation now runs every hour. On 21 October, 19 of 22 merges were recorded, and each gap was explained. One led to a simple rule: record first, then merge.

Both findings point the same way. A control is only as good as its coverage, and coverage is something to measure, not assume.

Why the MeshQu gate isn't approving merges yet

The automated merge gate is running, but in practice mode. It records what it would have decided next to what a person actually decided. Until that track record exists, every merge is authorised by a person: one by one, or under a standing rule that person set in advance. We are building the track record before letting the gate approve anything.

Two things this week told us not to hurry.

Reviews miss things. After some merges, we ran independent audits: a separate check, blind to the earlier reviews. In 2 of the 3 changes audited, the audit found real defects that several rounds of review had not caught. Both were in our internal build tooling, not the product, and neither reached staging or production. A small sample, but a clear signal. A policy that clears more work automatically has to be tested against what independent checks find, not only against what reviewers approved.

Automation quietly drops commitments. Twice this week, a practice we had agreed simply stopped happening: once the post-merge audits, once the logging of the merge gate's practice verdicts. Nobody decided to stop them. They lapsed. The analysis of the record is what noticed.

That second point is the one we keep coming back to. Agents are good at doing what is in front of them. They are less reliable at keeping up a practice nobody is checking. A record that is examined regularly is how a lapse gets seen.

What comes next

On 26 October we plan to compare the merge gate's practice verdicts with the decisions people actually made. Three things decide whether any check leaves shadow mode, and which one first: whether every checkpoint is being recorded, how closely the gate agrees with people's decisions, and what independent audits find. Any change will be deliberate and human-approved.

We are also watching whether quality holds as cost comes down. The cost controls are days old, so it is too early to say.

The pattern we are following is simple, and it is the one we would suggest to anyone putting controls around AI work:

1. Observe. Record decisions against policy without affecting the work.
2. Govern. Use the record to show what is unresolved, and who needs to decide.
3. Control. Let selected checks take part in the work, once the record has earned that trust, with a person deciding which checks and when.

The lesson for us was the difference between knowing our controls exist and knowing they work. We could measure how many merge decisions a revised policy might clear, question whether that was safe, find the evidence that was missing, and decide to keep a person in charge. That capability matters more than the 69.

The same questions come up well beyond software. A finance team approving a supplier, an insurer assessing a claim, an operations team releasing a payment: each needs to know which checks ran, what they relied on, and when a person had to decide. We have not tested those here. They are where we expect the same approach to apply.

This is an early account of internal use, not a study of productivity or control effectiveness. The figures come from our internal operational records, not from the published research corpus.

Where in your business are decisions still checked by hand, or made without a complete record? Bring one process to Qu and explore how policy, evidence and human judgement could work together.