Terminology

Access assertion

An Access assertion is the signed token Cloudflare Access attaches to every request that crosses the edge, as the Cf-Access-Jwt-Assertion header, after the requester signed in through Entra ID or presented a service token. The backend, the artifacts service and the arena verify it against the Access team’s certificates with their application’s audience tag and refuse a request without a valid one. The identity it carries is the requester’s: the email of a person, which is what a review records, or the common_name of a service token, which is how the CLI’s ingest and reference publishing and CI are identified. In-cluster callers never cross the edge and keep their service tokens and per-job tokens. See Identity.

Artifacts service

The artifacts service is the write-through store of every byte a run produces and every byte the public reads: a run’s playable build, its proof and asset media, its validation and showcase files and its recorded logs, in a private blob container per environment, and the public snapshot in the public bucket. The driver uploads each run’s tree to it, the console reads a run from it to play and review the run, and the backend writes the snapshot through it. It is stateless and the one writer of both object stores, so artifact bytes never transit the control plane.

Attachment pivot

A part’s attachment pivot is the point, in its parent’s local voxel coordinates, at which it hangs off its parent in a rig. Posing the parent moves the child about this point, so a turret’s pivot is where it sits on the chassis. For the root part the pivot is its origin in world space.

Backend

The backend is The Test Cabinet’s central private service. It is the canonical source of test case definitions for runners and the system of record for run results through storage, review, and publication. It verifies the Access assertion on every request that reaches it through the edge, and a service token on the mutating routes automation calls.

Bundle

A bundle is a run’s implementation as one file: the model’s work committed on top of the seed commit at the end of the run and written by git bundle --all. The driver uploads it to the repos service, which stores it, serves it for download and re-analysis, and pushes its main to the run’s public GitHub repository when the run publishes. The operator client’s installers are bundles in Tauri’s sense, one per platform; see Shell.

Catalog

The catalog is The Test Cabinet’s full set of test cases.

Climber

A climber is one combination enrolled on a ladder, the thing that actually does the climbing. Within a ladder’s dispatch each climber’s progress is tracked separately, and a climber is in one of four states:

• Running: working its current rung, which the dispatch can still launch.
• Blocked: its current rung is undecided and cannot progress until something is fixed, usually followed by a Retry. A dispatch with nothing in flight and only blocked climbers left to wait for reads Needs attention.
• Failed: it failed a rung and stopped there, so “failed at rung four” is a ladder’s headline result for one model.
• Completed: it passed every rung, and the ladder has no further question to ask of that combination.

Climber and combination name the same thing: the first is the role it plays on a ladder, the second is what it is.

Combination

A combination is the thing a run is executed by, as opposed to the test case it is executed on. It takes one of two shapes: a harness and model pair, plus a provider for a provider-routed harness; or a gg configuration and a model for each launch slot that configuration asks for. It is the unit a coverage plan crosses with its cases to form a cell, the unit a ladder enrolls as a climber, and the unit a reusable coverage group holds.

Console

The console is The Test Cabinet’s operator GUI, console.testcabinet.ai, a static web app served from the cluster through the tunnel behind Cloudflare Access. It mounts the routed application from the UI library. It enqueues runs at the backend, which a dispatcher drains into per-run driver Jobs, and it is both a reporter and a launcher of runs. It is same-origin with the services it calls: the backend under /api, the artifacts service under /artifacts and the arena under /arena. The operator client for the desktop and Android platforms is a shell that loads it.

Coverage

Coverage carries three meanings in The Test Cabinet, and they are not the same thing:

1. The measurement: how much of a declared matrix actually has runs. A coverage plan declares test cases pinned to a version, variant and engine, crossed with combinations and a target run count per cell, and its coverage is how many of those cells have met their target. A cell holds the first runs to land up to its target, and a run beyond that is not the plan’s. This is the narrower sense. See Coverage plans.
2. The feature area: the reviewer scheduling surface as a whole, which is plans, ladders, the reusable groups both draw their members from, the reviewer-wide runs-in-flight limit, and the halt and stop controls. This is the sense in which the console has a Coverage section and the backend a coverage API, and it takes in ladders, which are not plans and aim at no matrix at all.
3. Code coverage: how much of a produced implementation’s own src/ the tests the model wrote reached when they ran. It is measured by istanbul while the case’s [toolchain] test command runs, and recorded on the run record’s toolchain.test.coverage block.

So “a ladder is part of coverage” and “a ladder has no coverage target” are both true, in the first two senses. When it matters, say coverage plan for the first, the coverage surface for the second, and code coverage for the third.

Code coverage shares only the word with the other two. It is a property of one run’s produced tree, it measures the model’s own code with the tests the model wrote, and the test case’s validators contribute nothing to it because their own suite has coverage disabled. The static analyzer’s walk diagnostics on a run’s Code tab are headed Analysis notes rather than Coverage, since they describe the analysis and measure nothing about the code.

Debug API

A test suite’s debug API is the structured surface an implementation exposes so that validators can drive and observe it. The suite declares it as data, and an implementation publishes it either through a global handle or through the engine’s own debug object. Its functions are queries, which report state, and commands, which arrange the state a requirement is written against. A validator reaches an implementation only through this surface, which keeps its verdict independent of how the implementation is written. See debug APIs.

Dispatcher

The dispatcher is a thin, stateless controller that drains the backend’s run queue. It claims each queued run and creates one Kubernetes Job running a driver to execute it, and claims each queued publish and creates one publisher Job. The backend’s job table is the source of truth, so concurrency scales with the cluster.

Domain

A scoring domain is a facet of a test case the reviewer rates independently, such as a game’s single-player and versus modes. A case declares common domains that every variant is rated on, and a variant may add its own, so the effective set for a run is the common domains plus its variant’s. Each domain carries a functional rating, and the run’s functional rating is the worst across that effective set. A review item names the domains its failure lowers on a validator-rated version, and on a legacy version may roll up to a domain or stay general when it applies to every mode.

Draft

A draft is an editable, possibly incomplete state of a test suite that The Spec Cabinet authors. A suite holds any number of drafts, and a draft is saved whether or not it is complete. The Test Cabinet never ingests a draft; it runs the exported versions produced from one. See test suites.

Driver

The driver is the per-run executor: a one-shot process, created by the dispatcher as a Kubernetes Job, that runs exactly one test case. It holds the run engine. It resolves the definition from the backend, drives the run in an untrusted sandbox pod, streams live progress back to the backend, uploads the produced tree to the artifacts service and the implementation’s bundle to the repos service, and exits. The tcab CLI is its second binary.

Edge

Edge carries two meanings in The Test Cabinet:

1. The Cloudflare edge: where a request to a deployment arrives, is authenticated by Access and enters the tunnel. Nothing in a cluster is reachable from outside it except through the edge. See The edge.
2. A dependency edge: the one direction along which one repository of the set may depend on another. A build edge is one a manifest names; a data edge is one no manifest can name, such as the driver pinning gg’s released binary. See The edges.

Engine

In the context of The Test Cabinet, “engine” refers to two elements:

1. The runtime a produced game is built on, selected per run and recorded on it
2. The WebAssembly module a model submits for a performance test case

The first is provided to a run and the second is the deliverable of one. A performance case’s engine is the artifact under test, so it is never selected and a performance case supports no engine in the first sense.

Exported version

An exported version is an immutable, numbered snapshot of a draft that satisfies every test suite invariant. Exported versions are the suite states The Test Cabinet ingests and runs. See test suites.

Failure cap

A failure cap is a review item’s declared failure_cap: the highest functional rating the item’s domains may reach while the item’s validator fails, one of broken, scuffed, passable, or great. While the run’s effective checklist decides at least one scored item, a domain’s functional rating is the lowest cap among its failing items, so the caps let the validators decide the rating without a reviewer. A checklist that decides no scored item rates every domain broken. Every item of a validator-rated case version declares one.

Harness

In the context of The Test Cabinet, “harness” refers to two elements:

1. The Test Cabinet itself
2. Agentic harnesses used to drive models

The Test Cabinet runs other harnesses. It hits no LLM API and implements no agentic loop; that responsibility lies entirely with the agentic harnesses it drives.

Hub

The hub is contracts, the one repository every other repository of the set may depend on and the only place a shape two repositories exchange is defined. The set is a hub and its spokes: a library may be depended on, and an application, site, shell, tool-binary or content repository never is. See The dependency rule.

Joint

A joint is one named degree of freedom on a rig part: a rotation in radians about an axis through a pivot, or a translation in voxel units along an axis, bounded by a min/max/rest range. A caller-driven joint takes its value from a consuming game at runtime, for example turret_yaw. An auto joint is driven only by the model’s animation tracks and holds at its rest value otherwise. Joints are model-invented: a case declares the animations its rig must carry, and the model devises whatever joints carry them.

Ladder

A ladder is an ordered series of test cases that climbers ascend one rung at a time, stopping at the first rung they cannot clear, so the rung a model stops at is the result. Where a coverage plan asks whether a cell has been run yet and treats its cells as an unordered set, a ladder asks how far a model gets and treats its steps as a sequence in which each is harder than the last.

A ladder is a configuration and does nothing by itself. Running it starts a dispatch, which snapshots the configuration and climbs automatically: a single gate, parameterised by a rating floor and a threshold, reads the ratings the validators decided for the runs of each rung, and the backend moves a climber to the next rung as soon as the gate clears. A dispatch counts the runs that already exist for a rung, whoever launched them, and launches only the runs the rung is missing. A ladder keeps only its latest dispatch. Every rung is a validator-rated case version, and reviews never move a climb. See Ladders.

Leaderboard

Each test case has a per-variant leaderboard over its scored runs. Each row is one harness and model pairing, ranked by average score, then by rating, then by recency. gg runs are excluded, because a gg run’s agents may span several models.

Model

Models are the large language models that determine the actions an agentic harness takes.

Orchestrator

An orchestrator decides how a run’s harness sessions are conducted: how many sessions to drive, what each is told, and when the work is done. The harness still owns each individual session. An orchestrator is selected per run, is harness-agnostic, and defaults to one-shot, a single session. See Orchestrators.

Part

A part is one named voxel component of a rig, for example a tank’s chassis, turret, or barrel. Parts form a parent/child hierarchy, each attached to its parent at an attachment pivot, and each is sculpted independently with voxel-anim --part <name>. Posing a parent moves its children with it. Parts are model-invented: the model creates each part at run time with define-part.

Publishing

Publishing makes a run public. It releases the run’s playable build to Cloudflare Pages and its implementation to a public GitHub repository, adds the run’s document and media to the public snapshot, and mints the run’s short code. The backend enqueues a publish Job, the dispatcher admits it in the publish lane, and the per-publish tcab-publisher Job deploys the build and asks the repos service to push the bundle; the backend then writes the snapshot through the artifacts service, so a published run appears in the gallery with no site build.

A completed validator-rated run publishes itself the moment it completes, on its functional rating and score. A completed legacy run needs at least one review before it can be published, with two waivers. A run in a publishable failure state (catastrophic, timed out, harness error) has no checklist to complete. An auto-validated comparison run carries automated verdicts in place of a review.

A produced run is stored privately on the backend as soon as it finishes, and its build is playable for review off the artifacts service behind Access. Publishing is what first releases it publicly. See Results.

Rating

A run is rated on two channels, each a five-tier scale. The functional channel is rated per domain; the aesthetic channel is rated once for the whole run.

The functional rating says how faithfully a domain implements the spec: flawless, great, passable, scuffed, or broken, and a run’s is the worst across its domains. On a validator-rated run the validators decide each domain’s rating from its failing items’ failure caps, and rates every domain broken when its effective checklist decides no scored point; a review that overrides verdicts recomputes those ratings for itself, and a run with reviews takes the worst effective rating across them. On a legacy run each review carries one per domain and the run’s is the worst across every review.

The aesthetic rating says how the build looks, sounds, and feels to play: legendary, amazing, good, okay, or slop. amazing is the normal maximum and legendary is reserved for an exceptionally beautiful build. Only a validator-rated run has one; each review carries a single run-wide tier and the run’s is the worst across every review. See Evaluation.

Reporters

Reporters are The Test Cabinet components capable of reporting run results. Only GUI reporters allow users to interact with test case implementations. The console is both a reporter and a launcher of runs.

Repos service

The repos service is the write-through store of each run’s implementation as a git bundle, in a private blob container of its own. It serves a bundle for download, re-analysis and review, and on publish creates the run’s public GitHub repository, scrubbed with the shared redaction, pushes the bundle’s main and returns the URL, idempotently.

Requirement

A requirement is one RFC 2119 statement within a specification, classified as functional or non-functional. A functional requirement names one or more validators that decide it, while a non-functional requirement is judged by a human. Behavioral requirements are functional and appearance requirements are non-functional. See specifications.

Review

A review is a person’s assessment of a run after playing its build, providing the feedback automation cannot. Reviews are subjective, since games do not map cleanly to a rigid grading scale.

A review carries a prose writeup, a rating, and the identity the reviewer’s Access assertion carries. On a validator-rated run the rating is a single run-wide aesthetic tier, and the review may also override individual reviewer-checklist verdicts the validators decided. On a legacy run the rating is the functional one per domain, the review also carries a verdict on each reviewer-checklist item the case declares, and those verdicts and item weights produce the review’s numeric score, averaged across the run’s reviews. A run may carry one review per identity, typically from people other than the operator who produced it.

Reviewer checklist

A test case may declare a reviewer checklist: a list of major, observable requirements that every reviewer verifies by playing the build. Each item carries a point weight. An item may break into name-only sub-items, each judged on its own, with the item’s weight split evenly across them. A game-jam case grades its items on a five-level scale instead of pass/fail.

On a validator-rated run the validators decide every verdict, and a review may override any point’s verdict with a binary pass or fail and an optional note. Points a review leaves untouched keep the validators’ verdicts, so a review’s effective checklist is the validators’ verdicts overlaid with its overrides. Overriding is the exception, for a validator whose precondition could not be met or a build that does the right thing despite broken instrumentation. On a legacy run the console presents the checklist as a guided review with a completeness gate: every item and sub-item needs a verdict before a review can be saved or the run published. The checklist is reporter-side and is never seeded, so it stays out of the model’s input.

Rig

A rig is the posable structure of a voxel-animation model: its named parts in a hierarchy, the named joints a consuming game drives, and the model-authored animations those joints carry. The rig is model-invented: a case’s [model] table declares the required animations by name, and the model devises whatever parts and joints carry them. The produced rig.json carries everything the model built, and the voxel-runtime poses it for both the review viewer and real games.

Run records

A run record is produced each time a test case runs to completion. It records all information from the run, such as its run time, version information, and token and cost data.

Rung

A rung is one step of a ladder: exactly one test case, pinned to an exact version, variant, and engine, with an optional override of how many runs it takes to judge. The rungs’ order is the climb. Each rung carries a stable opaque id rather than being identified by its position, because rungs get reordered and every job a dispatch launches references that id.

A climber passes or fails each rung it reaches. A failed rung is the result the gate computed from the validator ratings of that rung’s runs, and the validators are assumed correct, so it stands as the climber’s result for that version of the case. An infrastructure failure or a canceled run never fails a rung, because neither says anything about the model. A run that ended on the model’s own failure, such as a timeout or a harness error, counts as a broken run.

One climber on one rung is a rung slot, the unit a ladder’s rung counts are reported in: three climbers on four rungs are twelve rung slots.

Runners

A runner is the component that actually executes a test case. There is exactly one: the per-run driver a dispatcher creates for each run. The console enqueues a run at the backend and watches it.

Runs-in-flight limit

The runs-in-flight limit is how many of a coverage plan’s or a ladder dispatch’s own jobs may be queued, pending, dispatched, starting, running, validating or storing at once. It exists so one plan or ladder shares the global queue fairly with everything else waiting on it. It is a reviewer-wide setting overridable per plan and per ladder, and is either a bound or no limit, which launches every missing run at once. Completed runs never occupy it.

Score

A score is earned points over the points available. Each reviewer-checklist item is worth a weight, a pass earns that weight, and a fail earns none, so the total is the sum of every declared item’s weight. An item with sub-items earns the fraction of its weight whose sub-items passed, so an earned score can be fractional. A validator-rated run scores from its validators’ verdicts the moment it completes; each of its reviews scores from its effective checklist, and a run with reviews scores the average of them. An effective checklist that decides no scored point scores zero over the declared weight. A legacy run scores per review and a run carrying several reviews scores the average. The run’s score is shown alongside its functional rating, and is what the per-case leaderboard ranks on.

A comparison scores a run from its automated validators alone, restricting both the earned points and the available points to the machine-checkable ones.

Shell

Shell carries two meanings in The Test Cabinet:

1. The operator client, client-tauri: a Tauri workspace, the shell kind of repository, whose thin native shell loads the console’s URL in a WebView and builds a bundle for Windows, Linux, macOS and Android. It links nothing of the project and signs in through Access inside the WebView.
2. The gallery application shell in @clockwyrks/ui: the routed application, its chrome and section navigation, which the gallery and the console both mount.

Short code

A short code is the handle a run’s share links are built from: the shortest prefix of the run id, at least eight characters, that no published run already holds. A run gets one the first time it publishes, stored on its record and as the object codes/<code> in the public bucket naming the run id, and it is never reassigned. The tcab.ai Worker resolves /r/<code> to the run’s verdict page and /p/<code> to its play page; see Serving.

Snapshot

A snapshot is the public dataset the backend composes from its published results and writes through the artifacts service into the public bucket: run documents, media and content-addressed index objects, with the index.json pointer swapped atomically after each publish. The static public gallery reads the index objects at runtime, so a publish is an upload and the gallery keeps no live dependency on the backend.

Specification

A specification is one folder of a test suite holding a natural language explanation of the behavior it covers plus the requirements that state it precisely. Specification folders nest, and every file in one belongs to that specification alone. See specifications.

Superrepo

The superrepo is the-test-cabinet: the repository that holds every repository of the set as a submodule at its root, with the documentation site, the issue board, the repository kit and the gates over the whole set. Each repository has a kind, which decides its shape: library (a Rust workspace with no lock committed, and an npm workspace), application (a Rust binary crate that commits its lock, and an npm workspace), tool (a library workspace whose binaries are also baked into the run-container images), shell (a Tauri workspace building desktop and Android bundles), site (an npm workspace and no Rust) and content (neither). See Repositories.

Test case

Test cases provide the scenarios used for testing. Each test case represents an isolated task that a harness and model must perform.

Test suite

A test suite is one authored project, committed in the test suites repository, from which one or more test cases are drawn. It is authored as drafts, and The Test Cabinet runs the exported versions produced from them. It holds the specifications an implementation is written against, the validators that decide them, the debug API those validators drive and read, the demonstrations, the reference implementations, the showcase, and the assets. The Spec Cabinet authors test suites and The Test Cabinet executes the test cases they define. See test suites.

Tunnel

A tunnel is the outbound Cloudflare Tunnel a cloudflared Deployment in each namespace holds, through which an environment’s one public hostname reaches its in-cluster services by path: / the console, /api/ the backend, /artifacts/ the artifacts service and /arena/ the arena, with Grafana on a hostname of its own. A cluster exposes no Ingress, load balancer, public IP or certificate, and TLS terminates at Cloudflare. See The edge.

Validation

Automated validation checks everything a machine can check: that an implementation builds and loads, how closely a view matches its reference image, and, through the instrumentation a case requires the build to expose, whether the spelled-out mechanics work when the build is driven into the states that exercise them. A build that fails the mandated debug-API contract fails automatically. Each validator’s pass or fail is visible per checklist point on a run’s Verdict tab, to every visitor including the public gallery. A game’s feel and quality are left to a human review.

Validator

A validator is a function that decides one requirement and returns an assertion result per condition it checks. It drives and observes an implementation only through the debug API, and a test suite holds one validator project. Each validator is claimed by exactly one requirement. See validators.

Validator-rated

A case version is validator-rated when it is on the engine-supported manifest format, the [workspaces] / engines / [[engine]] spelling of its starter project, and is not a game jam. A run is validator-rated when its case version is. Its validators decide its functional rating and score at completion, so it is publishable with no review, and those figures stand while the run has none, including the broken rating of a run whose validators decided no scored point. Its reviewers supply a single run-wide aesthetic rating and may override individual checklist verdicts, and a run with reviews averages their effective scores and takes their worst effective functional rating. Every other version is a legacy version, whose reviewers supply the functional rating and checklist verdicts.

Variant

Test cases may define multiple variants, which identify modifications to make to the specifications provided as input for the test. These variants may change game mechanics, add or remove content, and may noticeably affect the difficulty of a test case.

Voxel

A voxel is a single opaque-#rrggbb cell in a 3D grid, the 3D counterpart of a pixel. The two 3D asset-generation kinds sculpt into a fixed voxel volume, which always starts empty: voxel-model produces a static model, and voxel-animation produces a rig.

A voxel run’s authoritative output is the data its voxel binary emits: the meshed geometry as a per-part .glb, and a rendered preview. The validator parses and validates that emitted data rather than regenerating it, and the frontend renders an interactive 3D model with three.js.

Write-through store

A write-through store is a service that owns an object store and is the one writer of it: every byte reaches the store through the service and is read back through it, by a stream or a redirect to a short-lived signed URL, the service’s choice per route. The artifacts service is the write-through store of run trees and of the public snapshot, over a private blob container and the public bucket, and the repos service is the write-through store of run bundles. Neither keeps a volume, so both are stateless and scale horizontally.