Capability deep-dive · model production

A model reached production last quarter. Can you name its owner, and what it was tested against?

Most model-risk questions are not about the mathematics. They are about whether the chain that put a version behind a decision left anything behind — a dataset with provenance, an evaluation that actually ran, an accountable name, and a way back. A dashboard does not answer that. A refusal does.

Five steps, one job, one causal order

step 1

prep

prepared dataset — dataset lineage required before anything reads it

step 2

tune

checkpoint — submitted as a batch job through one gateway; streamed, not polled blind

step 3

eval-gate

eval report — the only step that can stop the promotion

step 4

register

registered model — a named owner, or no registration

step 5

deploy

serving endpoint — only a model that actually registered

There is no pipeline definition file. The chain is one row in the skill registry, and the five steps run as a single job with one identity, one progress stream and one causal order — so the thread for a training run is an unbroken chain rather than five loosely correlated jobs someone has to correlate afterwards. Each step’s output is stored under its own key and injected into the next by declared path, which makes the dependency graph a readable artefact instead of an implicit convention.

5chain steps registered and bound, each calling its executor directly with no model in the invocation path[O]◆

11compute providers registered behind the one gateway every training and evaluation job goes through[O]◆

6 × 3alert types by severity in the drift taxonomy that already exists, with queries and an acknowledge path[O]◆

no statisticbehind the drift types — the detection logic is not built — [P: to be calibrated][P]⧖

The platform today can carry, route, rank and acknowledge a drift alert. It cannot yet compute one, and an empty chart would be a worse answer than this sentence.

A model may argue inside a governed chain. It may never be how a step is invoked

This rule was not derived from principle. It was measured. Every step originally invoked its tool through a reasoning loop, and the model did not reliably emit the tool call — even with the call forced — so the executor never ran, the step output was an empty string, and the downstream steps fell into a skip cascade that read to an operator as “the evaluation report is empty”.

A LIVENESS FAILURE PRESENTING AS A CONTENT FAILURE

That is the general shape, and it is why the invocation seam is deterministic now. A step whose invocation is probabilistic has a liveness failure that arrives dressed as a content problem. Under log-or-refuse, an unexecuted step must be a refusal event with a code — and a probabilistic invoker cannot guarantee it produces one, because it may also fail to emit the refusal. Determinism at the invocation seam is what makes the run ledger’s silence mean something.

A second measured defect is worth carrying for the same reason: a queue name off by one value produced a dispatch that resolved, enqueued and returned success onto a queue no worker consumed. The run sat queued forever while everything else processed normally. A dispatch acknowledgement is not evidence that work happened. Only the run ledger is.

The promotion gate, and the hole in it we are closing

Registration runs only if the evaluation shows no quality regression and no safety regression, and deployment runs only if registration actually happened. That gate is expressed in the step definition and enforced in the executor, which short-circuits before the registry is even consulted — defence in depth that exists because the team hit the invocation defect above and did not want the gate to depend on a prompt.

The hole is precise and we would rather print it than have a model-risk reviewer find it: a skipped evaluation — meaning no evaluation ran at all — currently admits promotion. For a general-purpose factory that is a defensible default, because it avoids fabricating a benchmark that never happened. For an insurance factory it is inadmissible, because “we did not test” is not a pass under any validation regime. The required gate separates absence of evidence from evidence of no regression, and it is a strict narrowing of a shipped predicate: one branch removed, one comparison parameterised against the tenant’s own risk appetite.

AN UNREGISTRABLE MODEL CANNOT DEPLOY

No named owner, no registration. No registration, no endpoint. The chain is the control rather than a checklist somebody signs, which is a mechanism a model-risk reviewer can test in an afternoon. The assurance rung a version reaches — unevaluated, evaluated, cross-checked, board-approved, refused — decides what it is allowed to do, and a model may be cross-checked and still not touch a filed rating chain.

the five step executors(shipped)the single accelerated-compute path(shipped)the model plane of record(shipped)the evaluation harness’s rendered-script path(designed, not built)dataset lineage, the validation calendar and the model change board(designed, not built)

Where the chain fails closed

REFUSED

AUTHORITY_EXCEEDED

A promotion to a filed decision class was attempted without a sealed ruling from the named owner. The version was not registered for that class and no endpoint was cut over.

Promotion is a change to decision behaviour, so it is a governed decision in its own right rather than a deploy button — proposal, deliberation, cross-model gate, ruling with mandatory reasons, and a rollback pointer to the previous serving version, all on one thread. That is what makes a regulator’s treatment of a new model version as a capital event mechanisable instead of aspirational. Two of the refusals behind this are already enforced in code: a deploy step will not serve a model whose registration status is anything but registered, and a registration will not proceed on a reported regression. The board ruling and the assurance rung above them are designed.

the model change board and its sealed ruling — designed, not built

One more, stated because it is our own open item rather than a feature: where a training job is tenant-owned and no organisation resolves from the authenticated request, the chain must refuse with a machine code rather than substitute a platform identity. Today the seam substitutes. Making it refuse is a named build item, and it is a platform mechanism — every future application inherits it without its author knowing the factory exists.

The honest limit

WHAT THIS DOES NOT DO YET

Drift-detection logic. The alert taxonomy and stores exist; the detection does not.
Automated hyper-parameter search and architecture search are not built.
Tabular and gradient-boosted actuarial models are not built. The chain today produces and serves open-weight language models, and a rating chain is a different object.
Regulatory model documentation and reason codes are designed, not shipped.
The evaluation harness has a dispatch path but no rendered-script path, so the branch that produces a real evaluation is reachable in dispatch and not yet in execution. The entire promotion gate rests on it, which makes it the first build item rather than a footnote.
A skipped evaluation currently admits promotion. The narrowed gate that separates “no evidence” from “no regression” is designed, not built [P: to be calibrated].