For the Model Risk Lead
Every year the validation dossier is rebuilt from memory, spreadsheets and people who have since left.
The runs happened. The judgements were made, argued over, and acted on. But they were made in notebooks, in chat threads and in a version of the model that has since been replaced — so once a year a team reconstructs, from what survived, an account of work nobody can now re-run. The dossier is not the expensive part. The reconstruction is.
The run ledger a validator actually asks for
| Run | Model version | Inputs | Re-run | Judgement captured | Lands in |
|---|---|---|---|---|---|
| back-test vs experience | external cat model · version pinned | seed pinned sha256:2ae7f0…91cb | VALID | caveats recorded by a named reviewer, at the time | testing against experience |
| champion vs challenger | challenger version · champion version | seed pinned sha256:70b1cc…d43a | VALID | promotion thresholds and the decision not to promote | validation tooling justified |
| sensitivity and stability sweep | external cat model · version pinned | seed pinned sha256:c481d2…6f07 | VALID | rationale for the tools chosen over the alternatives | validation tooling justified |
| steered counterfactual | external cat model · version pinned | seed pinned sha256:5f9a3b…10de | VALID | unmodelled perils named, with the reasoning | model completeness |
| re-run after a vendor version bump | external cat model · version CHANGED | seed pinned sha256:5f9a3b…10de | MISMATCH | held: the change is a capital event, not a software update | model change governed |
Nothing in that table is written for the dossier. It is what the runs left behind on the day they were run, because a run without a seed and a pinned version is not admissible evidence and the platform will not record it as though it were.
[O] Illustrative composition, built from the components the product ships. No customer data appears anywhere on this site; the digests above are placeholders. The seeded correlated Monte-Carlo engine, the stress-scenario twin and the model-lifecycle plane are observed at ROS/src/services/gtm-genesis/blueprint/MonteCarloEngine.ts, ROS/src/services/DigitalTwinService.ts and ROS/src/routes/gpu-ml.ts.
REFUSED
HASH_MISMATCH
The re-run did not reproduce the sealed result, because the vendor model version moved underneath it. The row is held rather than quietly refreshed.
A validation ledger whose rows silently update to the current version is worse than no ledger: it makes last year's evidence look like this year's. The mismatch names the field that changed, the version it changed from and to, and the runs downstream of it — which is the input the model-change process needs, and the point at which a new vendor version stops being a software update and becomes a capital event.
The mechanism, in three lines
The annual validation dossier compiles itself, out of runs that were reproducible on the day they happened — rather than being reconstructed once a year from memory and spreadsheets.
First
Evidence is produced on the day
Every run is seeded and pure, its inputs are hashed, and the expert judgement attached to it is captured by the person who made it, at the moment they made it — not recalled eleven months later.
Then
Champion and challenger are one record
The comparison, the promotion thresholds and the decision — including the decision not to promote — sit in the same governed thread, so “why is this still the champion?” has a written answer.
Finally
The dossier is compiled, not written
Per model, per year, out of runs that already exist and can still be re-run bit-for-bit. What is left for a human is the judgement a human is supposed to be signing.
What compiles, and what you still have to sign
| What is asked, per external model, per year | Compiles from the ledger | Stays yours to sign |
|---|---|---|
| Demonstrate understanding of the model, its expert judgements, parameters and data | the judgements and parameters are logged where they were made | the understanding itself |
| Name the unmodelled elements — secondary perils, non-modelled territories, loss amplification | the counterfactual runs and their steering are recorded | the naming, and why the list is complete |
| Test against experience every year, with documented justification where it fails | the back-tests, their seeds and their re-run verdicts | the remediation plan |
| Justify the validation tooling chosen over the alternatives | the sensitivity, stability and benchmark runs actually performed | the argument for the choice |
| Govern model change: a new vendor version is a capital event | the version pin, the mismatch detection and the change record | the board decision |
THE PART WE DO NOT AUTOMATE
Licensing an external model does not outsource responsibility for it, and neither does licensing this. The right-hand column stays a human judgement on purpose — what changes is that the person signing it is reading evidence that was produced under the conditions they are being asked to attest to.
shipped(shipped)drift detection — designed, not built(designed, not built)
ROS/src/services/gtm-genesis/blueprint/MonteCarloEngine.ts is seeded and pure, and a run without a seed and a pinned model version is refused rather than stored as evidence. None of these is a customer measurement — we have no customers yet, and no figure on this page claims a saving.One model, one validation year, inside your own sandbox. If the ledger does not survive your own re-run, there is nothing here to buy — and that is a cheaper thing to find out in eight weeks than in an audit.
The honest limit
WHAT THIS DOES NOT DO YET
- Drift-detection logic. The alert taxonomy and stores exist; the detection does not.