v0.6.3 (2026-08-16)

The v0.6.3 release follows v0.6.2 and its centre is the reviewer’s loop: what gets run, in what order, and how much of it is allowed to pile up waiting for a human.

A coverage plan used to fire every run it was missing the moment you asked it to, in one fixed order, and then had no way to stop except cancelling jobs one at a time. It now emits in an order you choose, keeps a bounded review buffer rather than the whole matrix, refills itself as you review, and can be paused or halted with the cost of each control made explicit. Alongside it is a new entity, the ladder: an ordered climb that each harness+model combination ascends until a gate stops it, so “how far does this model get?” is a question the cabinet can answer on its own.

Underneath both, a job now records the account and the plan or ladder that launched it, which is what makes a scoped halt possible at all. The Runs page gains the global counterparts, and reviewing a run from a plan or ladder now returns you to the plan or ladder.

Away from that loop, the model detail page was rebuilt around a rule it had been quietly breaking: it now reports a model one test case at a time, and places it against the other models that ran the same one.

The run listings were rebuilt around the same question. They gain a filter bar that can name a case and a model and a version at once, an unpublished run now sorts among the published ones instead of being pinned ahead of them, and the whole selection lives in the URL. Beneath them, three defects in what the console was actually being served are fixed: the queue dispatched in an arbitrary order, two listings silently stopped draining at 200 rows, and the catalog was drained whole at app boot on every route. The services now compress what they send, and a publish re-uploads only the runs whose documents changed.

Those listings also stop guessing. The console holds one stream open to the backend and is told when a run is enqueued, claimed, changes phase, or ends, so a list that used to be a poll away from the truth now moves as the cabinet does. The runs that never made it out get a tab of their own, and a publish that failed finally says so.

This remains pre-1.0 software.

Features

A coverage plan emits in an order you choose

A plan’s runs execute in the order they are enqueued. Each job takes a monotonic queue position and the dispatcher claims strictly in ascending order, so the sequence a plan emits its cells in is the sequence they run in. That property had to be repaired first, and is below; what was never available is a say in the order itself. Every plan finished one case across every model before starting the next case, which is the right default and the wrong answer whenever the question is about a model rather than a case.

Plans now carry an outerAxis. “One case at a time” (the default, and the previous behaviour) finishes a case across every combination first; “One model at a time” takes one combination through the whole plan first. Nothing in the dispatcher, the driver, or the queue changed, because choosing the emission order is the entire mechanism, which is why the setting is exactly one field and no scheduling component learned about it.

Neither label mentions depth- or breadth-first. The choice is about what you want to be able to review side by side, and the traversal framing invites the reader to reason about a tree that is not there.

Plans hold a review buffer instead of firing the whole matrix

Triggering a plan’s missing runs enqueued all of them. That spends the entire budget before a single run has been looked at, and the first review is usually what tells you the plan was wrong: the wrong variant, a stale version pin, a model that never gets off the title screen.

A plan now keeps a bounded review buffer. Outstanding is everything in flight plus everything finished that you have not reviewed, and a top-up walks the plan’s cells in its own order, skips the ones already at target, and enqueues whole cells until the buffer is full. It overshoots by up to one cell on purpose: a cell’s repeats are the unit of judgement, so five runs of one case on one model are worth keeping adjacent in the queue even at the cost of running slightly over.

The target lives on the account, set under Settings → Reviewing, with a nullable per-plan override where null means “inherit” and 0 means “never”. How much work you want waiting on you is a property of the reviewer rather than of any one plan, and not of the plan list it was first bolted above. Topping up is an endpoint the console calls rather than a daemon: it fires when you open a dashboard, when you press the button, and, when autoTopUp is on, when you submit a review, which is precisely when a slot frees. It is serialized per plan by a claim on the plan row, so two console tabs cannot both notice the same shortfall and both enqueue for it.

Counting stayed global throughout: a run someone else produced still satisfies a target and is never re-requested. Only the judgement half is per-account. See Coverage plans.

Ladders

A coverage plan asks “have I run this yet?” and treats its cells as an unordered set. The new ladder asks “how far does this model get?” and treats its steps as a sequence: an ordered list of rungs (one version-pinned case each) that harness+model climbers ascend until one stops them, so the rung a model walls at is the result.

Whether a climber advances is decided by a single parameterised rule rather than a menu of modes:

advance when count(my runs on this rung rated FLOOR or better) >= THRESHOLD

Two knobs cover what reviewers actually ask for. At five runs per rung, floor scuffed with a 0.5 fraction is “stop when over half are broken”; floor scuffed with a count of 1 is “stop only when all of them are broken”; floor passable with a count of 1 is “one genuinely decent run carries the rung”.

The details that keep it honest:

• The gate reads only your review of a run. A run’s stored rating is the worst domain across every reviewer, so gating on it would let a stranger wall a climb that is not theirs.
• A run whose build never loaded counts as broken without waiting for a review (on by default). There is nothing to play, and making a human say so both stalls the climb and holds a buffer slot.
• A failed or canceled job is never a wall. Infrastructure failures retry; only completed runs are evidence.
• earlyStop is off by default, so a rung finishes all of its runs even when the verdict is already certain. The runs are evidence as much as they are a gate.
• Progress is stored per combination, never as one ladder-wide pointer, so a model added to a standing ladder starts at rung one while the others carry on.
• Both directions are manual-overridable, with hold to stop a climber where it stands and promote to advance past a wall, and an override is stored beside the automatic verdict rather than over it, so a recompute can never silently undo it and clearing it restores exactly what the gate says.
• Every verdict records the case version it was decided against, so bumping a rung re-opens it rather than silently inheriting evidence from a different spec.

A ladder is created disabled, and enabling it is the gesture that starts the spending. A plan tops up when you open its dashboard; a ladder doing the same would queue its first rung the moment you landed on the board you had just created it from, and again on every later visit. So opening a ladder’s board is a read, and the three gestures that feed one are enabling it, pressing Top up now, and submitting a review, which is precisely when a rung’s verdict can change.

Disabling writes only the flag: runs already queued carry on, and halt is still what cancels them. The switch lives beside the board that shows what it would start rather than in the editor, as the plan editor’s already does. The control that begins spending belongs with the evidence of what it spends on.

The same reasoning withholds an automatic retry while the plan or ladder that launched the run is paused. A failed run re-queueing itself minutes after the reviewer stopped the queue is the same surprise as an unasked-for top-up, with nothing in the console that appears to have caused it. A run launched by hand has no plan speaking for it and retries as before.

The editor is a column of settings rows: the name and what it controls on the left, the control on the right, and a reset beside any value that differs from where the form loaded, which replaces the “on by default” captions the page used to carry.

Rungs drag into place on a grip handle, since their order is the only thing that makes a ladder a ladder and nudging one a step at a time was the only way to build a climb easiest-first; the ▲▼ buttons stay for the precise single step and for reordering without a pointer. A drag announces the case and the position it would land at, so a keyboard lift says what the sliding rows say to everyone else. Reordering stays local to the draft, and the save still reconciles rungs on their stable ids, so a rung that merely moved keeps every climber’s verdicts.

Performance and game-jam cases are refused as rungs with an explicit error: neither can ever produce a rating for the gate to read, so a rung holding one would stall a climb while looking perfectly healthy. See Ladders.

Pause, halt, and halt all

Stopping a plan meant cancelling its jobs by hand. There are now three controls, and they are separated by what they cost rather than by how forceful they sound. Pause stops topping up and leaves the queue alone. Halt additionally cancels the plan’s queued and pending jobs, which have no driver and have spent nothing, so it needs no confirmation and is the common case. Halt all also reaches dispatched, starting, and running, discards work that is partly paid for, and is confirmed rather than defaulted to.

Every halt reports how many jobs it cancelled. A halt that answered only “OK” cannot be told apart from a halt whose scope was wrong, and those two call for opposite next moves.

Ladders carry the identical three, though the pause switch is surfaced there as Enable/Disable: a ladder that is not climbing has usually not been started yet rather than been interrupted, and enabling it is what asks for the first runs. Both reuse the existing atomic cancel transition rather than introducing a second state machine for bulk work.

A job records who launched it, and why

Jobs gained two nullable columns: user_id (the launching account, which was already resolved on every launch and simply discarded) and origin (plan:<id> / ladder:<id>, null for a launch by hand). The prefix matters because plan and ladder ids are minted independently, so a bare id would let one halt the other’s runs.

This is what makes a scoped halt safe: a run launched by hand from the new-run form is never swept up by a plan’s halt, by construction. An automatic retry inherits the original launcher and origin rather than the retrier’s, so a retried run stays in the buffer that asked for it and remains reachable by that plan’s halt.

Coverage counting deliberately ignores both columns. Counts stay global; attribution exists for halting, not for bookkeeping.

Global stop controls on the Runs page

The trailing edge of the Runs/Failures/Unreviewed tab bar now carries Clear pending, Kill active, and Stop all, backed by three bulk-cancel endpoints named after the job states they sweep rather than after the buttons. They are global, stopping the cabinet rather than one plan, so narrowing them by account was rejected outright: it would silently skip every job recorded before jobs carried an account.

Each reports its count and the states it reached, so the console phrases “stopped 12 runs, including 3 already executing” from the response rather than from which button was pressed. Below the medium breakpoint the cluster drops to its own full-width row beneath the tab strip rather than collapsing into a menu. These are the controls you reach for when runs are burning money, and putting Stop all behind an extra tap is the wrong trade.

Reviewing a plan’s runs returns you to the plan

A plan or ladder dashboard exists to be worked through, and every run opened from one used to land you back on the global run list when you pressed back, which threw away the position you were reviewing from.

A plan’s cells link to their runs, and a ladder’s rungs expand to list theirs inline in the shared run log rather than sending you to a pre-filtered Runs page: a rung’s verdict is an argument about its runs, and answering it should not cost the reviewer the board they were reading. Wherever a run is opened from, the dashboard registers itself as the place to return to, so open → review → back lands where you started.

Both surfaces also expose their own unreviewed queue, in the plan’s or ladder’s emission order rather than newest-first: the buffer was filled deliberately, and walking it in the order it was filled is what keeps a cell’s repeats comparable against each other. On a ladder it matters more still, because the review is the verdict.

pending is surfaced as its own state

A plan reporting a full buffer with nothing apparently running was indistinguishable from a wedged queue. Cells and roll-ups now report pending separately from inFlight (of which it is a subset): a job sits pending when its harness is at its parallelism cap, or when it is a game jam run of a model that already has one in flight, which are both the queue working exactly as designed.

The console is told about a run instead of asking

The console learned about in-flight runs by polling every worker’s active list every 15 seconds. A push feed already existed, but it carried only terminal alerts. It announced no enqueue and no phase change, so a run started by a coverage plan, a ladder, another tab, or another machine was invisible until a poll happened to find it, and a tracked run’s phase advanced on the timer rather than when it moved.

GET /notifications is now a multiplexed stream with two topics, every frame a named SSE event: a stream hello frame carrying the connection’s id, then notification (alerts, on by default) and run (lifecycle events, off by default). PUT /notifications/{stream}/topics is the control channel SSE does not have. The console holds one stream open for the session and flips the run topic on and off as it enters and leaves the pages that show in-flight runs, rather than opening a second stream per page and dropping the alerts riding the first.

The split is between alerting and list maintenance, which are not the same event. An alert is something a person is shown, so it fires only for what is worth interrupting them over. A run event is every transition a list must reflect, including the many nobody wants a toast for: a run held back to pending, a driver reaching starting, forty runs ending at once under a bulk sweep. A cancellation is exactly the case that needs one and not the other. Both topics ride one channel so their order is guaranteed, and a run leaves the list before its completion toasts.

Backend-side, every site that moves a job publishes the event: enqueue (single and batch), claim, the driver’s status reports, and both cancel paths. claim_next now reports the queued/pending transitions it reconciles around the job it claims, which were previously invisible to everything outside the transaction.

The poll is gone rather than merely slowed, which took closing the two ways an EventSource fails silently. After enough failed attempts it settles on CLOSED and the browser stops retrying for good. A half-open socket, such as a resumed laptop or a dropped NAT flow, delivers nothing while still claiming to be connected. The stream emits its own heartbeat event so the second is detectable at all, because keep_alive writes SSE comments, which EventSource consumes internally and never surfaces to the page.

The transport arms a watchdog on every frame and reopens, with backoff, when one is overdue or immediately when the browser has given up. A third failure, a client lagged on a healthy connection that the handler used to skip in silence, now reports itself as a resync frame.

Nothing is replayed for a topic that was off, so turning the run topic on asks for a resync: the “fetch on navigate” that the live events are an update stream on top of. Re-basing buffers events for the duration of that fetch and replays them over the result, because the snapshot describes the queue as it was when the request was served, and applying it flat would undo what the stream had already reported by re-adding a run that finished moments earlier. The old poll swept such a row away within an interval; with no poll it would have stayed for the session. See the console stream.

The publish backlog has a tab, and a failed publish says so

Publishing is asynchronous by design, enqueuing the release and moving straight on to the next run, but nothing announced a release that never landed. A transient GitHub 5xx recorded its reason on the publish job row and closed a live stream the console had almost certainly navigated away from; the run then sat unpublished, indistinguishable in the all-runs listing from one nobody had got round to. Publish failures were effectively invisible.

Two halves fix that. The feed gains a second alert kind, publish-failed, raised on a terminal publisher result and carrying its reason. It is keyed by the publish job rather than the run: the run has usually already raised its own “Run complete” alert under its record id, so keying by the run would dedupe against that, and a second failed attempt would silently overwrite the first instead of alerting again. Raising it is best-effort, since the durable failure is already recorded by the time it fires.

The console switches on the kind, because the two mean different things about the same run: a completion prunes the in-flight list, while a failed publish changes neither the list nor the run, which finished long ago and is simply still unpublished.

The other half is /runs/unpublished, a fourth tab beside Failures and Unreviewed: the runs that have cleared the publish gate but have not been released, rendered as the same dense run log with the same filter bar, so a backlog narrows to one case or model. Its rows are selectable, so a whole batch is published from the log’s right-click menu. It is console-only, like its sibling worklists, since the public gallery holds nothing unpublished by definition.

The slice is a new backend state=publishable, which mirrors the publish gate as a query rather than restating it. A database test asks the gate itself about each seeded run and compares the two sets, so they cannot drift. It is deliberately narrower than the existing unpublished slice: a worklist whose whole purpose is “select these and publish them” must not list the unreviewed runs and the never-publishable infrastructure failures the publish endpoint is about to refuse.

A batch enqueues each release and stops there rather than awaiting it, which would hold a live stream open per selected run for minutes and pin you to the page. A refused gate still surfaces immediately, and a release that starts and then fails now arrives as a notification. See the runs section.

A model page reports one test case at a time

A model’s detail page could tell you what the model is and list what it had run, but it could not answer the question the page exists for: how well has this model actually done? The nearest thing was a reliability ring over every run the model had ever recorded, which pools a sprite case and a full end-to-end game into one figure and so describes the mix of cases that happened to be run rather than the model.

The detail page now opens on an Overview tab built on the opposite rule: figures from different test cases are never mixed. It picks one case and variant, a dropdown each, listing exactly the cohorts the model has runs for, with the run count beside each so an unrepresentative one is visible before you switch to it. It then reports that cohort alone: runs, completion rate, mean reviewer score, mean cost, mean tokens, and mean run time, with an outcome ring and a ratings ring beneath. The version scope control the case tabs carry is here too, anchored on the versions this model has runs for so the default scope always selects runs that exist.

Beneath that, the model is placed against the field: the share of the other models that ran the same case and variant whose mean it sits above, for cost, tokens, and reviewer score (“more expensive than 78% of the 9 other models…”). Each model in the field is reduced to its own mean first, so a model that was run fifty times does not count fifty times, and the comparison is only ever taken within one cohort, because a percentile over mixed cases is a confident number that means nothing.

The selected cohort lives in the URL (?case=, ?variant=), so a particular reading is a link.

The About and Pricing tabs are gone

Two of the model detail tabs were carrying nothing. Pricing charted a model’s price history, but a model’s prices change rarely enough that the chart was almost always two or three points. The history is still recorded, still what a run’s comparable cost is priced against, and the current per-Mtok rates still show on Stats; only the chart and its table are gone. About showed the curated description, which is still edited on the model config form but no longer earns a tab of its own.

The detail page’s tabs are now Overview, Stats, and Runs, with Overview as the index, so an existing /models/<id> link still lands somewhere useful. /models/<id>/pricing no longer resolves.

A run listing filters on more than one axis at once

Every run listing offered exactly one filter: a free-text q, one substring OR’d across the recorded case, model, harness, and variant. That cannot express “this case AND this model”, cannot address a test-case version at all, and on a model’s Runs tab was not offered in the first place.

The all-runs index, a model’s Runs tab, and a case’s or jam’s Runs tab now share one filter bar: the search field, the equality facets the route does not already pin (test case, version, harness, model), and a “Current versions only” toggle that is on by default. Every facet is a server-side equality filter, so the facets AND with each other and with the search rather than competing with it, and the whole selection rides in the URL (?case=, ?version=, ?harness=, ?model=, ?latest=0), so a narrowed listing is a link.

The toggle exists because a case version is frozen once it has runs, so an older major.minor is a different spec whose runs are not comparable with the current one’s. Which version is “current” is resolved from the runs in the queried lifecycle slice rather than from the definition store: a newly authored version then cannot blank a listing before anything has run against it, and the static gallery, which has no store and only its run index, answers the identical question from the same data, so both hosts page identically. Naming an exact version overrides the toggle, since asking for an older version must show it rather than silently emptying the listing.

GET /runs?fields=summary gained version and latestVersions to serve this, the latter resolving its per-case allowlist from one SELECT DISTINCT (slug, version) shared by the listing’s count and its page. The console’s two version parsers collapsed into one module whose ordering key mirrors the Rust catalog’s version_key, so v1.10.0 sorts after v1.9.0 on both sides of the wire. See the web console.

Every question the console asks is the cabinet’s own

Every destructive control asked its question through the browser’s confirm(), an OS chrome box that reads as nothing to do with the cabinet and can carry only a line of plain text, and two batch failures were reported through alert(). All seventeen call sites now go through a themed modal: run delete and kill, the runs sweeps, plan and ladder halts, group/plan/ladder/model-configuration deletes, rung bumps, mark unplayable, and the reviewer’s bulk verdict restore.

Underneath are a Dialog primitive (scrim and neon-outlined panel portalled to the body; Escape and scrim dismiss, focus opened on the default action and trapped, the page behind locked) and a useConfirm() provider whose promise-returning confirm(…) / alert(…) keep each call site the guard-clause shape it already had.

The dialog is capped at the viewport height with an optional detail region that scrolls inside it, so it can enumerate arbitrarily much without outgrowing the page. “Restore validator verdicts” uses that to list every point it would change and which way each verdict would flip: a count alone does not tell a reviewer who has just worked through the whole checklist whether it is the correction they meant. See Dialogs.

A wrong address renders a page that says so

The app’s <Routes> had no catch-all, so a path no route matched rendered nothing but the top bar over an empty main element. That reads as a broken site rather than as a wrong address, and it is what every mistyped URL, dead link, and stale bookmark got.

The catch-all page names the status, echoes the path that was asked for so a typo is visible, and offers the sections to go to instead, mirroring what the top bar shows on each host. Being last in <Routes> it also catches the patterns that exist but are gated off the current host, such as /runs/new on the gallery and the whole account section, which is why that gating can stay in each section’s router.

The route table is also exported, with its own matcher, through a dependency-free @test-cabinet/ui/routes subpath, for a server that has to tell a real page from a wrong address before the app runs. Nothing consumes it in this release; see the short-link deferral.

The services compress what they send

Nothing the console fetched was compressed. The backend, the artifact service, and the console’s own nginx all served text verbatim, which is invisible on a fast link and brutal on a thin one. GET /runs hands back whole run records and already reaches ~2.3MB on prod, so a reviewer behind a VPN waited out a payload the backend itself had produced in ~57ms: the delivery was the cost, not the work.

All three now gzip. On the backend the CompressionLayer is outermost so it sees every route’s final response, and its default predicate is what makes a service-wide layer safe: it skips sub-32-byte bodies, images, gRPC, and text/event-stream. That last exclusion is load-bearing here, since compressing the live streams (/jobs/{id}/live, /notifications, /runs/{id}/events) would buffer events behind the encoder and stall every live view.

The artifact service adds application/gzip to that predicate, because archive.tar.gz is served pre-compressed and re-encoding it spends CPU to return larger bytes. nginx’s gzip defaults to off, so the console had been shipping its hashed bundles verbatim on every first load.

This is negotiated, so it is not a behaviour change for existing callers: the workspace pins reqwest without its gzip feature, so the driver, CLI, dispatcher, and publisher never advertise it and keep receiving byte-identical uncompressed responses. Browsers advertise it and get the benefit.

A publish re-uploads only the runs that changed

The snapshot refresher rebuilt and re-uploaded one JSON document per published run on every publish, so publishing a single new run re-exported the entire corpus and the cost of a refresh grew without bound as runs accumulated.

Run and case media already avoided this by living outside any one snapshot’s prefix and being referenced rather than re-uploaded; the documents, which are the bulk of the set, now get the same treatment. A per-run document moves from snapshots/<id>/runs/<run-id>.json to a content-addressed documents/runs/<run-id>/<digest>.json.

A document is not immutable, since a new review or an arriving proof rewrites it, so the digest is over its own bytes: an unchanged run lands on the key it already occupies and uploads nothing, and any change mints a new key, with no separate “has this changed?” signal to go stale. Uploads are issued with bounded concurrency rather than one round trip at a time; the only ordering the snapshot needs is the index.json barrier.

The key is not derivable from the run id, so each runs.json summary names its own documentKey and index.json drops runsPrefix for the snapshot-independent runDocumentsPrefix. That is snapshot schema version 2: a consumer reading per-run documents must follow documentKey rather than composing a path. Superseded revisions are deliberately left in place. The grace period an orphan needs runs from supersession, while a listing only reports creation time, so pruning on that clock would delete a just-orphaned document and 404 a site build still reading the previous generation.

Two more full scans found in the same audit are gone: the summary pass indexed the case catalog once instead of scanning it per run (it was runs × cases), and normalize_free_model_ids pushes its :tag predicate into SQL instead of loading every run row, record_json included, and filtering in Rust. See the public snapshot.

Fixes

The queue runs in the order it was filled

A batch of repeated runs started, and so finished, in an arbitrary order, so three runs each of three cases arrived interleaved instead of a case at a time.

The claim ordered by (created_at, id), and neither key orders a batch. POST /jobs/batch stamps one identical created_at on every row it inserts, so the whole batch was a total tie that collapsed onto the tiebreaker, id, a random UUIDv4. created_at is also stored as a string whose RFC 3339 subsecond part is variable-length, so even distinct timestamps do not always compare chronologically (…:00.55Z sorts before …:00.5Z), which could mis-order separately submitted runs too.

Each job now takes a monotonic queue_seq minted at insert, and the claim orders by it. A batch takes a contiguous block of positions in the order it was submitted, inside one transaction, so both consoles’ case-major fan-out survives into the queue: the new-run form’s runs-each, and a coverage plan’s per-cell missing runs. The active-run list orders by it too, so it reads in the order the runs will start.

Game-jam serialization and the per-harness parallelism caps are unchanged. They still gate which waiting job is claimable, and only the order of the scan moved. This is the property the whole of coverage’s emission order rests on. See queue order.

A produced run sorts with the published ones

Every run listing pinned the console’s produced runs, which are unpublished and so unreviewed, ahead of the queried page, so those rows ignored the active sort entirely: a header sort re-queried the server for published runs only and then prepended the locally-held worklist on top of the result. The case and jam Runs tab went further and drained the case’s whole run set to filter, sort, and page it in the browser.

The summary listing gains a state=any slice, the union of published and unpublished, plus a variant equality filter, and every listing draws from it. Each page is now one filtered, sorted, windowed server query, so an unreviewed run takes its place in the same order as everything else and only a page of rows is ever fetched. Only in-progress runs, which have no record to list yet, still lead the first page. The static gallery answers any from its in-memory index, so both hosts stay behaviour-identical.

A case’s and a model’s runs are drained completely

The case-scoped and model-scoped drains asked for limit: 1000 and advanced offset += 1000, but the backend silently clamps a listing to 200 rows. Past 200 runs the offset outran the data: the first window returned rows 0–199, the next asked for offset 1000, rows 200–999 were never fetched, and the loop then terminated on the empty result. The case leaderboard, the case metrics, and the model stats had been ranking and averaging over a punctured set.

Both drains now advance by the number of rows that arrived, which is correct against any host free to return a short page, and request the backend’s actual ceiling rather than a limit it will not honour. The regression test fails against the old offset arithmetic.

The console fetches a case’s detail when you open it

The console drained the entire catalog before it could render anything. fetchTestCases resolved every version of every case, 176 stored versions here, and then, for each case’s latest version, every variant’s rendered spec bodies. That is several hundred requests and megabytes of manifests, none of which a listing shows. The catalog fetch also lives in the root gallery provider, so the whole drain was paid at app boot on every route, including the runs list, the models page, and a deep-linked run, for grids that show a name, a difficulty, a summary, and some tags.

The catalog contract is now split in two. GET /test-cases carries each case’s display metadata, read server-side from its latest visible version, and every list surface renders from that one request; a case whose latest manifest cannot be read is skipped rather than failing the listing.

The detail is fetched per slug for the one case a visitor opens: description, variants with their prompts, seeded specs, references and checklists, changelog, errata, domains, sheet, and model. It goes through a hook that caches the in-flight promise per (resolver, slug) so the several detail surfaces that mount together for one case share a single fetch. The static site holds the whole snapshot in memory and serves both halves from the same inlined array.

A fetch in flight reads as a wait, not a dead end

Several surfaces reported their data unavailable while the fetch that would supply it was still in flight. The run detail Inputs tab was the worst: useRunVariant discarded the catalog’s status and returned undefined both while the catalog was loading and when it genuinely held no such case, and the tab turned that single undefined into “The inputs for this run’s test case are not available.” Since the catalog is fetched independently of the run record, that message was what a visitor normally saw first.

Every site below now distinguishes three states, where a fetch in flight is a wait, an unreachable source is a fault, and only a settled, empty result is genuinely unavailable: the Inputs tab, the Models list (which claimed the catalog was empty mid-fetch and said nothing at all when it failed), a case’s Leaderboard tab (both boards), and a case’s Metrics tab (which called the sample too small to chart while the runs it would chart were still arriving).

Five more pages that rendered a bare “Loading…” paragraph now show the arcade loading mark every other page body uses: the unreviewed worklist, the all-runs list, the tournament detail page, and the account Groups and Coverage plans tabs. The Events tab keeps its text-plus-progress-bar treatment, because it streams a potentially large file and reports real transfer progress, which an indeterminate mark cannot express.

A catastrophic run gets its own slice of the reliability ring

RunState::Catastrophic is publishable and, in its own words, “reported as a separate catastrophic-failure statistic”. The model exited cleanly but produced nothing that builds or loads, which is real signal at the benchmark’s edge. The model page’s reliability ring had no segment for it, so those runs fell into the widget’s uncolored remainder: no legend row, no tally, and four legend numbers that did not sum to the total in the middle of the ring. A model that produced nothing runnable a third of the time simply looked like it had fewer runs.

The widget documented that remainder as intended behaviour, which is what kept it from reading as a bug. The one state that legitimately has no slice is infrastructure, which is the Test Cabinet’s own failure, never publishable, and excluded from every model statistic. The ring now carries a catastrophic segment and orders its segments as RunState::ALL enumerates them, so it cannot drift out of step with the contract again.

A deep link is served at its own URL

A deep link such as /runs/<id> has no file behind it, so the gallery carried a public/404.html that stashed the requested path in sessionStorage and bounced through / to restore it client-side. That is the GitHub Pages workaround, which exists because GH Pages has no rewrite configuration at all, and it was carried over unnoticed when the gallery moved to Cloudflare Pages, along with a comment that restated a GH Pages limitation as a property of static hosting in general.

The cost was not only the visible flash: every run page answered crawlers with an HTTP 404 carrying no markup, so a run link resolved to an error when shared and no run was indexable. Nothing surfaced it because nothing linked individual runs around yet.

Deleting the file is the entire fix. Cloudflare Pages serves /index.html for an unmatched path, at the requested URL, with a 200, whenever a project ships no 404.html. The /* /index.html 200 rule tried first is rejected by Pages as an infinite loop, since it strips /index.html back to /, which re-matches /*, and it was never in effect. apps/site/public/_redirects is kept with no rules, as the place a reader looking for Pages configuration will find out why there are none, and why not to add that rule back. The stray public/CNAME, GitHub Pages residue from the same move, is dropped.

A live run is not thrown away to shrink the cluster

A driver Job is backoffLimit: 0, so an evicted driver pod is never replaced. It is a destroyed run, along with the model spend that run had already incurred. The cluster autoscaler cannot infer that: it sees a controller-backed Job pod and treats it as freely relocatable. The driver is also an unusually attractive scale-down target, because it requests a tenth of a CPU while the sandbox holding the run’s real reservation is a separate pod, frequently on a separate node, so a node whose only tenant is a driver sits below the utilization threshold for the whole length of the run. Three runs were lost to this in 48 hours.

Every pod the dispatcher and driver create is now pinned with cluster-autoscaler.kubernetes.io/safe-to-evict: "false", the sandbox included. The autoscaler spares controller-less pods today, but that is a property of cluster configuration rather than of the manifest, and it inverts the moment anything gives the sandbox an owner.

The death report is corrected with it, because it called this “failed before its pod started” and sent the reader looking at admission instead of at the cluster. The pod’s DisruptionTarget condition is read ahead of its container state, so an eviction is not masked by the SIGTERM the container reports (exit 143), which describes how it died rather than why. When the pod is gone entirely the Job’s status.failed count, which outlives it, tells “never started a pod” from “ran and was deleted mid-run” instead of asserting the former. See the dispatcher.

An upload no longer builds the whole archive in memory

Two services materialized an entire run archive in memory, and each did it at the worst possible moment.

The driver tarred a finished run tree into a buffer before uploading it. Measured against recorded container metrics, that path peaked at 476MiB while every other moment in a driver’s life cost 3–9MiB, and it happens after the harness session has ended and before terminal status is posted, so a driver killed there loses the most expensive thing in the system.

upload_run_tree now builds the archive into an unnamed temporary file on the driver’s own out-dir and streams it back through a ReaderStream. The out-dir is the volume already sized to hold a run tree and shares its filesystem; a /tmp that turns out to be a small tmpfs would fail halfway and, being memory-backed, reinstate the very allocation this removes. The file is unnamed so it has no directory entry, which lets the kernel reclaim it when the descriptor closes, including when the driver is killed mid-upload, and no error path has to remember to clean up.

Tarring moves to the blocking pool, since the same task is still relaying harness events and heartbeating status, and the body stays chunked: a wrapped stream has no size hint, and declaring a Content-Length alongside chunked framing is the one error here that truncates an upload silently rather than loudly.

The artifact service took the other end of that upload as Bytes, holding the whole tarball resident before a single entry was unpacked, up to the 2GiB cap. That is the one allocation a long-lived shared service must not make, because it is the only component here whose peak is set by a caller’s payload rather than by its own work, so its ceiling could not be sized from anything it does.

upload now takes the raw body and spools it to an unnamed file in the store root as it arrives, then unpacks from there. That file sits on the same filesystem as the run trees, so the archive and the tree it becomes draw on one pool of free space. The body is touched only after the job token is verified, and the cap is applied per chunk while writing rather than read off a Content-Length a chunked request does not carry, which also means it now bounds disk rather than memory. The DefaultBodyLimit layer goes with it: it constrains only the buffering extractors and is inert against a raw body, so leaving it would have been a cap in name only.

The service’s two read paths (tree.tar, archive.tar.gz) still build their archive in memory. They are bounded by a run tree rather than by the upload cap, but they are the same shape as the bug fixed here. The publish Job’s resources also become configurable the way the driver’s already are, via TCAB_DISPATCHER_PUBLISHER_*.

Smaller fixes

• The Other section’s game jam and tournament detail pages shipped without the back chevron every other detail page carries, so a visitor who opened one had no way back to the list beside the browser’s own back button. Both now render it, and the Other page records its viewed tab so back returns to Game Jams or Tournaments rather than the section default. On the tournament page the subject line (case, version, variant) moves up to sit beside the chevron, and is rendered in every state, including while loading.
• The review rail’s “Mark unplayable” and “Restore validator verdicts” shrink-wrapped to their labels, so they sat narrower than the category rows beneath them and the rail read as ragged. Both stretch to the rail’s width.
• A tooltip raised by tapping a chart bar on a phone could not be dismissed. Plot raises the tip from the pointermove a tap delivers but clears it only on a mouse pointerleave, and because the charts point by column with a wide radius, tapping elsewhere in the chart merely moved the tip to another bar, so there was no gesture that took it down. <Chart>, the single place a Plot figure is mounted, now hands Plot the mouse-shaped pointerleave it is waiting for when a non-mouse tap lands outside the figure. Mouse behaviour is untouched: hover-out still dismisses, and a click-stuck tip still unsticks on the next click.
• The run filter bar’s facet dropdowns sat at their own content width, leaving the row visibly short of the full-width search field above it. They now share out the row’s spare width, wrapping on narrow viewports.
• The snapshot contract page claimed a run’s aggregate score “is not summarized here; the site computes it client-side”. That has not been true since the summary card gained score. The real distinction is that the score is not readable from the run record alone, because the point weights live in the case catalog, and that is now what the page says. It is the difference between a consumer reading every headline figure out of runs.json and believing it has to walk a document per run.

Development

The scoring rules move out of the UI package

The rules that turn a review into a number lived in @test-cabinet/ui’s ratings module, which makes them unreachable to anything that must not depend on React, such as a build step, a worker, or anything computing a Test Cabinet figure outside a browser bundle. They are also already mirrored per function in the Rust core, so a second TypeScript copy written elsewhere would be a third implementation to keep in lockstep.

They now live in @test-cabinet/run-stats: zero runtime dependencies, types only from the run-record contract. ratings re-exports the scoring half, so every existing import keeps working, and what it still owns is the display metadata (labels, emoji, prose) that is presentation. GRADE_META reads its point values from the package’s GRADE_POINTS rather than restating them, so the scale that scores a run and the scale shown beside it cannot disagree.

The package also adds rollupRuns, which reduces a set of RunSummary cards to one set of figures, so a figure frozen at one moment and the same figure recomputed later are produced by the same function rather than by two that drift. It reduces the summary index, so a rollup costs one already-cached fetch and never walks per-run documents.

Token and cost figures are optional per run, since not reporting a cost is distinct from a run being free, so each is summed as a metric carrying its reported and unknown counts rather than as a bare total that would present a partial sum as a complete one. Score is offered both pooled by weight and meaned per run, since conflating the two is exactly how a frozen figure and a live one stop being comparable. The model Stats page takes its reliability breakdown from the rollup, so the shared function is the real code path rather than dead code.

The front-end tests run in CI

Neither CI system ran the front-end unit tests at all, so every vitest suite in the repo was green only on whichever machine last thought to run it. They now run on both, as a web-test.sh job of their own beside the build rather than as steps inside it: a failing assertion should report as a failing test rather than as a failing step named “Build website”. The two also need different things, since the tests need only the small workspace runtime packages built and never the app bundles, so they run in parallel.

The ordering is load-bearing. run-record, run-stats, and the two runtimes publish their entry points from a built dist/, so on a clean checkout, which is exactly what CI is, a suite that imports one fails to collect. A root build:packages script builds them first, and build:site is redefined in terms of it so that dependency order lives in one place rather than two that drift.

That drift had already happened: web-build.sh built the gallery against its own inlined list of prerequisite packages, which had gone stale, so CI built the site against two packages it had never built and the leg failed for reasons unrelated to the change under test. It calls npm run build:site now, the same script Cloudflare’s git-connected gallery build runs, so CI exercises the production build path rather than an approximation of it. The site is still built with no TCAB_SNAPSHOT_URL, which is the empty-dataset path a fresh deployment takes and now the only place it is exercised, making it the regression guard for a temporal-dead-zone crash that broke exactly that build.

The console image builds the package the console now imports

The web image’s build stage installs the workspace and builds the console, but @test-cabinet/run-stats, new here where the scoring rules moved, was neither in the build context nor built ahead of it. Its exports map points every subpath at compiled output, so the packages/ui sources the console bundles lost both the re-exported scoring rules and the rollup, and the type build failed on both architectures. It is now included and built after run-record, whose types it needs, alongside the two runtimes that are built there for the same reason.

No tracked text file hides from grep

A raw NUL byte makes every tool that reads text decide a file is binary. grep and rg then report zero matches in it rather than an error, so its contents are not merely awkward to search; they are silently invisible to every search anyone runs.

Two tracked files carried one. useModelRunSummaries.ts joined and split its model-id cache key on a literal NUL written into the string literals, which took the whole hook out of every grep of packages/ui; it is written as a \u0000 escape now, the same character at run time and ordinary text on disk. The other was a stale .pyc that .gitignore already covers, so it is untracked rather than escaped.

The gate that keeps them out cannot ask git whether a file is binary, because git infers binary from a NUL byte, and under that rule every file the gate exists to catch excuses itself. The binary formats this repository commits are declared in .gitattributes and the hook reads that instead. Each extension there was verified wholly binary in this tree before being declared, which also stops git trying to diff and merge a wasm component or a rendered video as text.

The actions move off Node 20

GitHub now forces node20-based actions onto Node 24 and warns about it on every run. The last six actions still declaring runs.using: node20 are bumped (docker/login-action, docker/setup-buildx-action, docker/build-push-action, actions/upload-artifact, actions/download-artifact, actions/setup-node). No usage here touches the majors’ breaking changes.

Short links are deferred to v0.7.0

A short-link cluster was built during this cycle and taken back out before release: a tcab.ai resolver Worker, a share-links contract package, a gallery Pages Function that injected per-run preview tags, a console share control, and a shareBaseUrl advertisement on GET /config. None of it was deployed, with no DNS record, no Worker deploy, and no Pages build variables, so nothing regresses, and v0.6.3 does not ship a public client-config field nothing reads.

It comes out rather than shipping because the gallery is moving off the fully static design in v0.7.0: it becomes a server-routed origin reading a public projection of the published set, so it decides status codes and preview tags itself. That removes the two things the cluster existed to work around, a static host answering every path with the shell and a 200, and a <head> written before the run is known. It also replaces the derived short code with one persisted at publish, which is the only fix for the collision the derived design could not avoid. Shipping it now would mean releasing a surface that is reimplemented, and re-broken, one version later.

Three pieces are kept deliberately, because the new design needs them too: the not-found page and its route table, the deep-link behaviour the gallery is deployed with today, and the content-addressed run documents.

The schema gains five migrations

Upgrading to v0.6.3 applies five, all additive: queue_seq on job (a 0 default, so rows already queued when it runs stay valid and sort ahead of everything minted afterwards, which is correct because they are older), the job attribution columns, the coverage plan scheduling fields, the account-level coverage settings table, and the ladder tables.