# When a model explains a decision, what has it proved?

Canonical: https://www.meshqu.com/blog/explainability-is-not-proof

AI explainability shows how a model behaved. It does not show which policy, threshold and model version were in force when the decision was made.

Published: 2026-04-23T00:00:00.000Z

> AI systems are becoming more explainable.
>
> Being explainable is not the same as being provable.

A model declines a loan application at 14:02 on a Tuesday in March.

The model can account for itself. Feature importance scores, <Term definition="SHapley Additive exPlanations, a widely used method for assigning each input feature a numeric contribution to a model's output, drawn from cooperative game theory.">SHAP</Term> values and local explanations describe how the inputs moved the output.

Six months later, someone asks why that applicant was declined.

At that point the explanation answers a narrower question than the one being asked. Which policy applied at 14:02? Which threshold was in force? Which version of the model ran, and was it the version the model risk committee signed off? If a person overrode the outcome, did they have the authority to?

These are practical questions, and a description of model behaviour does not settle them.

AI explainability is often treated as the answer to AI governance. The argument runs: _if we can explain the decision, we can defend it; if we can defend it, we can prove it._ It fails at the first step. An explanation describes _how_ a model behaves. It does not establish that a decision was **made under the right conditions**.

## An explanation describes behaviour, not correctness

Take the same loan applicant from the [AML](/blog/the-problem-isnt-policy) and [underwriting](/blog/what-a-decision-looks-like) examples in the companion pieces. She submits at 14:02 on a Tuesday in March. The model declines. SHAP returns a weighted breakdown: income +0.3, debt −0.6, credit history −0.4.

That breakdown is useful, and it is about the model. It tells you how the output was produced, not whether the decision **should have been made**: whether the threshold the model used matched the policy in force at 14:02, whether the analyst's override authority applied, whether the model version running was the one the model risk committee signed off.

A SHAP plot does not carry the policy that applied, the thresholds in force, the context the decision was made in, the version of the model that ran, or whether the decision complied with governance at that moment. It describes behaviour, and correctness is a different question.

## Explanations drift, and they are not independent

An explanation is also tied to the present. Models are retrained, features are reweighted, pipelines change. Six months later, the same SHAP output cannot be reproduced with certainty against the same input. **The explanation drifts.**

It is generated by the system under question, too. The model explains its own output and the infrastructure describes its own behaviour, so what you hold is a system narrating its own decisions rather than evidence another party can check.

That is not the question a regulator asks. A regulator does not ask _how does your model work?_ They ask _why did this decision happen_, in one specific case, at one specific moment, under one specific policy. Explanations are descriptive. Proof is evidential.

## A provable decision is recorded once

A provable decision does not depend on an explanation produced after the fact. It is captured at the moment it is made, with the input, the policy, the context, the outcome, the version, the timestamp and the signature bound together as a single artefact.

Not inferred. Not recalculated. Not approximated.

When the question _why did this happen?_ comes, the answer is not generated. It is retrieved.

_Illustrative example: a generated receipt for the fictional applicant above, not a measured retrieval._

<ResolvedReceipt
  question="Why did the model decline this applicant on 12 March?"
  receiptId="RCP-MODEL-43821"
  resolvedIn="Illustrative retrieval"
/>

Be precise about what that settles. A retrieved record shows what was evaluated, which policy and version applied, and that the record has not changed since it was signed. It does not show that the inputs were true, that the policy was well designed, or that the outcome was the right one. Those remain questions for the people reviewing it.

An explanation helps you understand a model. A record is what lets someone else examine a decision.

Start with one decision that another person may reasonably challenge: a decline, an override, a flagged transaction. Capture the policy, the threshold, the version and the outcome at the moment it is made, and keep the [receipt](/receipts) that binds them together.