Essay

Explainability Is Not Proof

Sam Carter23 Apr 2026 · 3 min read

AI systems are becoming more explainable.

Being explainable is not the same as being provable.

The promise

When an AI model makes a decision, the expectation is simple: explain it.

Why was this customer declined? Why was this transaction flagged? Why did the model produce this outcome?

Tools now exist to answer those questions — feature importance scores, values, local explanations. The system can describe its own behaviour.

The assumption

AI explainability is often treated as the solution to AI governance. The argument runs: if we can explain the decision, we can defend it; if we can defend it, we can prove it.

Where it breaks

An explanation describes how a model behaves. It does not prove that a decision was made under the right conditions.

The gap

Take the same loan applicant from the AML and underwriting examples in the companion pieces. She submits at 14:02 on a Tuesday in March. The model declines. SHAP returns a weighted breakdown: income +0.3, debt −0.6, credit history −0.4. Useful — but it answers a different question.

It tells you how the model produced the output. It does not tell you whether the decision should have been made — whether the threshold the model used matched the policy in force at 14:02, whether the analyst's override authority applied, whether the model version running was the one signed off by the model risk committee.

What's missing

A SHAP plot does not capture the policy that applied, the thresholds that were in force, the context the decision was made in, the version of the model that ran, or whether the decision complied with governance at that moment.

The plot describes behaviour. It does not establish correctness.

The time problem

Even when an explanation exists, it is tied to the present. Models are retrained. Features are reweighted. Pipelines change.

Six months later, the same SHAP output cannot be reproduced with certainty against the same input. The explanation drifts.

The independence problem

The explanation is generated by the system itself. The model explains its own output. The infrastructure describes its own behaviour.

This is not independent evidence. What you have is a system narrating its own decisions.

The moment of scrutiny

A regulator does not ask how does your model work? They ask why did this decision happen — in one specific case, at one specific moment, under one specific policy.

The difference

Explanations are descriptive.

Proof is evidential.

The shift

Explanation describes. Proof evidences.

A provable decision does not rely on explanation after the fact. It is captured at the moment it is made, with the input, the policy, the context, the outcome, the version, the timestamp, and the signature bound together as a single artefact.

Not inferred. Not recalculated. Not approximated.

Recorded once.

What changes

When the question comes — why did this happen? — the answer is not generated. It is retrieved.

Asked

Why did the model decline this applicant on 12 March?

RCP-MODEL-43821 VerifiedResolved in 2.1 seconds

Closing

An explanation helps you understand a model.

It does not help you prove a decision.

When a decision needs to defend itself — in front of a regulator, a court, or a risk committee — understanding is not enough.

You need proof.