2026-09-15

The test that certified the outage

On our platform, the “Run now” button of an automation answered with a refusal for nearly six weeks, from 22 July to 31 August 2026, for every client. Scheduled runs, meanwhile, ran normally. Continuous integration stayed green throughout the outage, because the test covering that path checked precisely the faulty value.

A hard-coded string

The on-demand run sent the model router a model name hard-coded in the source, an alias that router has never served. The endpoint concerned forwarded that name verbatim, without resolving it the way the other endpoint does. Every run therefore died before its first token, on a model access refusal. We confirmed it across the whole platform by trying the key of an automation with 1,081 successful scheduled runs: it was refused too, on that path only.

Why the outage stayed invisible

Two reasons, whose conjunction explains the duration. Scheduled runs went through the system's scheduler, which resolves the configured model itself: the history kept filling on time, and monitoring saw nothing unusual. And the unit test for the on-demand run asserted exactly the hard-coded value: it checked that the code sent that very name. The test was green because the bug was in place.

The fix, and the corrected test

The code now reads the model name from the configuration, as the scheduler does. The test checks against that configuration, and refuses by name the two values that caused the outage. The path was replayed end to end on 31 August: trigger, run, report, run record.

The rule we keep

A test that pins a literal value proves the value, not the behaviour. When the value is wrong, it certifies the outage, and monitoring that looks elsewhere will not see it. Since then, a test of that kind checks against the source of truth, the configuration or the contract, and names the values that have already misled.

A test that pins a literal value proves the value, not the behaviour: for six weeks, ours certified the outage.