Methodology

Every number on this site comes from a file results/<id>/<date>.json that validates againstr1-bench.v1. The site is rebuilt from those files; a file that does not validate fails the build.

How a result is produced

  1. The implementation starts with AFORO_TEST_MODE=true and AFORO_CLOCK_FIXED=2027-03-01T12:00:00Z and loads the frozen seed seed-v1.json.
  2. The conformance suite runs black-box against it: Schemathesis (contract), Hurl (acceptance, one file per rule and error case), k6 (race invariants: 300 VUs on a 50-seat event), the security checks and the webhook receiver. Web implementations also run @aforo/e2e (28 Playwright scenarios, axe, Lighthouse CI).
  3. The load profile is fixed: 200 RPS for 3 minutes (70 % reads, 25 % availability, 5 % bookings). Latency percentiles, errors and memory are recorded.
  4. Each measurement runs 5 times on GitHub Actions ubuntu-24.04 (4 vCPU); the file holds the median. make bench reproduces it locally.
  5. Factory columns are aggregated from the implementation devlog with the factory-run.v1 schema.

Acceptance criterion

An implementation is conformant with 100 % Hurl, 0 Schemathesis failures, exact race invariants and no ZAP High alert. Its own line coverage must be at least 85 % (80 % for web).

Noise caveat

Shared CI runners are noisy: the same commit can move by 10 to 30 % between days. Runtime numbers are markedindicative. Read them as orders of magnitude and look at the history page of each implementation before drawing a conclusion. Conformance numbers are deterministic and are not affected.

Build order is a bias: later implementations reuse lessons from earlier ones. The factory columns carrybuildOrder so the comparison can account for it.

Factory columns (factory-run.v1)

Fields of factory-run.v1 shown on the scoreboard
FieldMeaning
tokensInput / tokensOutputInput and output tokens of the agent sessions that built the implementation (cache reads and creations kept apart).
costUsd, costBasisCost in USD, either `api-equivalent` (list price of the tokens) or `billed`.
wallClockMinutes, sessionsElapsed time and number of agent sessions.
humanInterventionsTimes a human had to step in.
defectsAfterMergeDefects found after the work was merged.
toolCallsTool calls made by the agents.
tokenSource, estimated`ccusage` when tokens were measured, `unavailable` otherwise; estimates are flagged.

Publishing a result

Add results/<id>/<YYYY-MM-DD>.json through a pull request to r1-aforo-spec. The id must be listed in results/registry.yaml, the folder must match the id and the file name the date.