Skip to main content
Make is the developer-facing command surface. Cargo is the default backend. Pull-request CI is Cargo-only on GitHub-hosted runners and selects its lanes from the changed paths. BuildBuddy/Bazel is an opt-in acceleration backend for developers with BuildBuddy access, and the whole BuildBuddy graph runs nightly and as the release-tag validation gate; it never runs in pull-request CI. Local Cargo-default behavior does not select the CI backend.

Local Commands

Use these commands for normal development:
scripts/affected-tests.mjs selects the tests a change affects with the CI planner; without --run it prints the commands instead of running them. Avoid raw cargo for normal repo work. When you need targeted Cargo commands, use ./scripts/repo-cargo so output roots and multi-agent work stay isolated. A crate’s own tests build with its test features: the features its [package.metadata.meerkat.standalone-tests] lists, plus workspace-tests when it declares that feature. No dev-dependency turns them on, so a test command passes them. node scripts/core-workspace-args.mjs --tests -p <crate> prints a crate’s selection with them, and --tests alone prints Core’s whole-workspace selection with every member’s. Run tests in that form:
The Make targets, the CI lanes and the Cargo aliases pass them already. In this repository a test build without them fails to compile, naming the exact command to run, instead of leaving the crate’s workspace-only tests out. Only code generation trips that check, so cargo check, clippy and an editor’s check run are unaffected. The check sits in the crate’s lib test, so a run narrowed to an integration target (--test <target>) is not stopped: without the flags it compiles the tests they gate out without a word. Pass them there too, plus any feature the target’s own #![cfg(feature = ...)] names (meerkat-core’s wholeblob_compaction_media_guard needs test-support; a second --features adds it):
For rust-analyzer, set rust-analyzer.cargo.features to "all": without features it treats the code the test features gate as inactive, and its Run test lens stops on the check. A package-qualified feature list does not work there, because Cargo refuses a feature of a package the command does not select and rust-analyzer’s runnables select one package. "all" also turns on every other feature (the live, realtime and GPT Live ones, and the products’), so the editor’s check compiles more than a default build does; it is still the one global value that works.

BuildBuddy Opt-In

Set MEERKAT_BUILDBUDDY=1 to route the same Make lanes through the BuildBuddy backend:
Use BUILDBUDDY_DRY_RUN=1 with explicit BuildBuddy Make targets to inspect the selected command without running it. The explicit targets are make buildbuddy-build, make buildbuddy-check, make buildbuddy-clippy, make buildbuddy-test, make buildbuddy-test-unit, make buildbuddy-test-int, make buildbuddy-e2e-fast, and make buildbuddy-e2e-system. The live-provider targets are make buildbuddy-e2e-live and make buildbuddy-e2e-smoke when credentials are configured. The Make targets delegate through scripts/run-build-backend-lane; when BuildBuddy is enabled, that switch calls scripts/buildbuddy-dev.

Test Lanes

The canonical e2e lane taxonomy lives in tests/integration/src/e2e_lanes.rs. Scripts and BuildBuddy targets route to that taxonomy rather than inventing separate lane names.

Public Live real-audio smoke

Turbo S scenario 98 remains the existing e2e_smoke_s98_gpt_live_public_playback_settlement_and_reopen selector:
This is a paid public gpt-live-1/OpenAI test, not a mock or a readiness probe. It requires OPENAI_API_KEY (or RKAT_OPENAI_API_KEY) with Live voice-session access, Node/npm, Chromium, and working provider WebRTC. Missing credentials, refused transport, and absent speech fail the selected test; none becomes a successful skip. The canonical lane installs the browser prerequisites. The existing browser peer sends checked-in synthetic WAV speech through a WebAudio media track, never a microphone or user recording. Every checked exchange requires new inbound RTP and measured non-silent decoded audio (at least 100 ms) or non-silent WebAudio analyser windows. RTP energy is diagnostic only; an absent browser statistic remains absent, never a measured zero or successful audio evidence. Signaling, transcript deltas, and provider completion events cannot satisfy that evidence gate. Numeric GPT_LIVE_PUBLIC_AUDIO records report measured audio. The scenario verifies exact close/reopen on the same durable session, with a five-second exact-close ceiling and a 90-second open/reconnect ceiling, spoken code-word recall after reopening, and real tool execution through the feature-owned ExistingMember background-delegation policy. Generated operation custody must identify the original member and report a completed turn; its canonical history must contain a successful pwd result linked to the exact shell tool call (zero exit, no timeout, and the actual project directory), followed by a provider assistant answer. Error-only tool results cannot satisfy this gate. No disposable fork is spawned. Its ordinary text model and session identity remain unchanged. S98 calls ServiceMemberLiveHost’s shared, exact-receipt lifecycle directly; S97 retains the RPC/durable-fork vertical. After the initial delegation completes, S98 commits a later ordinary background turn that changes the code word, then requires the unchanged live channel to speak that new word using fresh measured audio. This is separate from delivery of the initial delegation’s result. The existing executor model defaults to gpt-5.6-sol; an operator may select another available ordinary text model with GPT_LIVE_E2E_EXECUTOR_MODEL without changing the Live channel’s gpt-live-1. S98’s Bazel shard remains //:e2e_smoke_turbo_s_s98; both backends use the same Rust catalog and test. Scenario 99 is a separate opt-in paid acceptance scenario; S97/S98 retain their existing selectors and behavior:
The generated Bazel shard is //:e2e_smoke_turbo_s_s99; it carries the same public Live test binary and routes through the canonical Rust catalog. S99 composes LiveContextSummaryPolicy with Concurrent bootstrap and an externally gated, content-only summarizer held for at least 20 seconds. The gate carries permission only: after release, a separate factory-built LLM client summarizes the immutable source snapshot using its exact model/provider and auth binding. No canned summary is used in paid acceptance. The caller bounds this real call to 90 seconds and 1024 output tokens, disables caller and provider-native tools, and requires measured input/output token usage. S99 requires an exact-receipt Active owner, connected WebRTC, fresh provider output transcript honestly lacking an unpredictable historical phrase, and at least 100 ms of fresh decoded non-silent native voice before releasing the summary. Analyser-only activity, RTP, silence, or transcript alone cannot qualify S99. Checked-in synthetic WAVs are the only speech input; a zero-gain oscillator keeps the media track and RTP alive between speech fixtures. S99 selects ProviderManagedUnmeasured bookkeeping, with no output queue, polling, publication receipt pump, or manual playback completion. Its required binder publisher is a fail-closed guard against unexpected actionable playback publication; the owner must settle internally without calling it. Provider transcripts and decoded-audio assertions are evidence, never settlement signals. S98 retains its separate caller-confirmed playback path. While preparation remains Generating, an ordinary text update and a newer spoken correction are admitted, and an existing-member delegation must complete with the same successful pwd evidence as S98. After receipt-validated ProviderAcknowledged, fresh marked spoken queries must recall the historical phrase while retaining both newer facts over the old summary. Historical thinking-context delivery requests no speech, but releasing its barrier can legitimately release queued speakable work results. S99 neither suppresses those results nor asserts full-duplex silence. Exact thinking-append/ACK correlation and absence of fabricated user turns belong to deterministic provider/owner tests with attribution. Further close/reopen cycles attempt to release an obsolete job (or observe its cancellation) while a replacement job remains gated: the replacement must still report preparation and honestly lack the history until its own job is released. All waits are bounded (20-minute inner scenario, 25-minute catalog command). Missing auth is a failure, not a skip. Compilation and deterministic catalog/audio-evidence tests are not evidence that this paid scenario has passed; native-provider qualification requires explicit execution of the selector above. The dense Mob topology stress has a dedicated GitHub-hosted Linux workflow, .github/workflows/mob-dense-topology.yml. Nightly calls it (its archive build alone takes 13-17 minutes on a hosted runner, so it does not fit the pull-request budget), and the diagnostic Cargo workflow also runs it from the existing unit-test archive. The test owns a five-minute wire-materialization budget; its nextest profile reports the named test every minute and terminates a whole-test stall after eight minutes. The result is bounded to the hosted runner environment: local macOS and BuildBuddy runs are supplementary evidence, not latency predictions or substitutes for this lane.

Architecture Gates

Machine and surface contracts have direct Make targets:
These gates protect generated schemas, SDK wrappers, machine artifacts, and runtime authority invariants. make machine-verify is the bounded TLC lane used for normal validation. Use make machine-verify-full only when you deliberately need the unbounded verification set.

CI Shape

Pull-request CI is .github/workflows/ci.yml. It runs on pull requests, pushes to main and to release/** integration branches, and manual dispatches (any other branch head runs once, through its pull request), entirely with Cargo on GitHub-hosted ubuntu-24.04 runners (4 vCPU). A pull request runs what fits the owner’s cap of 20 minutes from the start of the pre-push hook to a green CI gate, and the gate also enforces 1080 s of lane execution and a 1200 s push-to-terminal ceiling on it, runner queue excluded; the push runs every lane. Compiled objects are cached by sccache (a release pinned by sha256, content-addressed and shared by every lane; push and schedule runs, and a dispatch on a release branch (rel/** or release/**, so it warms the bucket for that tree’s pull request and main push), set the client to write it, every other run, a pull request included, to only read it, a client-side setting, see Known limits); rust-cache holds only the registry, because per-lane target caches exceeded the repository’s 10 GB cache limit in one pull request: What runs where (pull_request is budgeted; push: main is not; a push to a release/** integration branch runs the same push-only lanes as main, so reverse-dependency suites run on every release merge, but only main writes the exact-tree attestation; nightly and the release tag carry the rest): Reader rows. Some tests read files that belong to no crate: the rkat doc-example test walks docs/, the machine-authority docs alignment test reads docs/ and .claude/skills/, xtask’s static lanes read CHANGELOG.md, and so on. scripts/ci-non-rust-readers.json lists every such input and its reader (scripts/ci-non-rust-readers-guard.mjs keeps it complete, and fails a reader whose named test its features compile out), and a change to one adds the exact target that reads it to the integration lane: -p the reader with the inventory’s features, its lib, bin, test or example target (the lib and bins for a reader in library or build-script code: its test targets, like those of a pull request’s code change, are the push’s closure-tests rows’, which also keep the nested-Cargo canaries) and a filter on the reading tests, where no other lane of the plan runs it. A pull request leaves the readers that compile meerkat-mob (named) and the readers keyed only by the whole checkout (live audits that every path selects) to the push. A push runs its readers’ test targets in closure-tests, which build each package with the features a whole-workspace build resolves, and every package’s lib and bins in main-unit; a test target that needs other features (such as xtask’s machine-authority contracts or the gpt-live-replay suite) gets a reader row with them, and a push with no main-unit (no Rust-relevant path, such as a docs-only push) gets reader rows for the lib, bin and example targets. A reader whose features need live-provider credentials (integration-real-tests, openai-live-e2e) runs in no CI lane: the plan names it (reader_not_run) and the CI gate prints it. Documentation inside a crate is no build input unless the crate compiles it in (include_str!/include_bytes! in library, binary or build-script code, such as the crates’ skills/*/SKILL.md); that selects the crate like a change to its code. Long rows first. clippy, main-unit, closure-tests and feature-check run in two waves, each wave a job of its own (<lane> and <lane>-rest). The rows that took 15 minutes or more on a green run, and rows never measured, start right after changes, as do the closure check’s per-product jobs; the shorter rows, and the short lanes (ratchets, integration, governed-jsonl, sdk-host, example-web, wasm-timers, bazel-graph), wait for the stagger job, which sleeps 30 seconds on a push and not at all on a pull request. The account’s hosted runners take about 40 jobs at a time and pick queued jobs in no reliable order, so on a whole-workspace push of about 170 jobs a long row picked late set the wall time. The measurements are scripts/ci-row-durations.json, regenerated from green runs with scripts/ci-row-durations.mjs --repository OWNER/REPO --output scripts/ci-row-durations.json RUN_ID...; a stale entry only moves a row between waves. A row is keyed by its name without the member count of a packed row (core+12 is core, openai+8 bazel set is openai bazel set), so a merge that moves line counts and repacks the rows keeps a row’s measurement while the bin’s heaviest member, which names it, stays the same. The CI gate counts a two-wave lane as run when every wave that holds rows succeeded. Other runs on the account (pull-request CI, nightly, dispatches) take runners from the same pool: keep them off while a release commit’s main run is in progress. “Deferred” lanes run on the push of the merge to main (or a release/** branch), whose CI must be green before a release tag. A pull request’s plan names them (deferred_to_push) and the gate prints them. They are deferred because each needs 9-30 minutes on a 4-vCPU runner, mostly to compile the meerkat-mob chain (meerkat-mob alone compiles for 5-11 minutes whenever it misses the cache), and the owner’s cap is 20 minutes from the start of the pre-push hook to a green CI gate on every pull request. How the budget is measured: the gate reads every completed lane from the Actions jobs API and splits its time into runner queue (created_at to started_at) and execution (started_at to completed_at). On pull requests it enforces two limits:
  • Lane execution, 1080 s (CI_MAX_EXEC_SECONDS): each lane’s own run time plus the run time of the change classification it waits for (and of the stagger job, for a lane that waits for it), over the slowest lane. This is the part the code controls: a lane that grows or an aggregate that drifts fails here. Run times belong to the job, so re-running only the gate cannot turn an overrun green.
  • Push-to-terminal excluding runner queue, 1200 s (CI_MAX_TERMINAL_SECONDS): the runaway backstop on the latest lane terminal minus the runner queue on that lane’s path (its own queue plus the change classification’s, and the stagger job’s for a lane that waits for it, in the same attempt; the jobs API creates a job when its needs complete, so a lane created after the stagger job completed waited for it). Each lane is timed from the start of the run attempt it ran in: a re-run lane gets a fresh clock, and a carried-over lane keeps its attempt’s clock. Past the ceiling a run’s jobs are stuck. Waiting for runners alone never fails it.
Both limits hold every lane a pull request plans for ordinary reasons. An exact-profile lane (governed-jsonl, mcp-confinement-positive) that the plan runs only because the change touches its inputs or changes its definition (GUARD_LANE_INPUTS, GUARD_LANE_DEFINITIONS; the plan exports the reason as guard_lane_reasons) is outside both: the confinement lane’s cold build alone takes 27.5-32.5 minutes. The gate still requires it to succeed, its own timeout-minutes (40 on a pull request) bounds it, and the gate names it, with its seconds and the reason, in a Guard lane outside the budget warning and the guard_lanes_outside_budget output. A guard lane run on every Rust change (a push, a dispatch, a release projection) or for an unknown diff stays in the budget. Runner queue wait is reported, not budgeted: at 300 s or more on the critical path (CI_QUEUE_WARN_SECONDS) the gate emits a CI runner queue wait warning. The account runs at most 40 hosted jobs at once, shared by every pull request, main push and nightly, and a busy hour fills that cap. The gate writes a typed verdict (within_budget, queue_delayed, exec_over_budget, runaway) with the execution path, terminal and queue seconds to its step outputs and summary, and prints the lane table on every run that was not cancelled, including runs whose lanes failed, pushes and dispatches. The budget keeps pull-request feedback fast; a push or a dispatch is a full validation run, where nothing is enforced and a failure to measure is only a warning. Sizing: the owner’s cap is 20 minutes from the start of the pre-push hook to a green CI gate, and the hook takes about a minute, so 1080 s (18 minutes) of execution leaves about a minute of slack and 1200 s (20 minutes) of terminal is the runaway backstop at the cap. On the workspace-mode pull-request runs 38859112297 and 38931641220 the lanes a pull request still plans ran at most 854 s (wasm-check; example-web 842 s cold) after 28-31 s of classification. The earlier budgets, 1500 s and 2700 s, were sized on 66 attempts of 2026-09-30 to 2026-10-01 (execution path max 1244 s) before the meerkat-mob chain’s and the closure’s lanes moved to the push. Why the meerkat-mob chain is deferred: measured on 4-vCPU hosted runners, every unit lane that ever exceeded the budget compiled meerkat-mob’s 442k lines, either as its own lane (16 min compile plus 4.5 min of tests) or as a dependent crate rebuilding mob under its own feature unification (rkat, rpc: 17 min). The chain is computed from cargo metadata, and the classifier’s cost model (1 + (lines + 8000 x dependency closure) / 35000 minutes, with a 20-minute surrogate packing cap per pull-request unit lane; the cap is a model threshold, not a measured timing) is asserted by the self-test for every other crate. Clippy of a changed mob-chain crate still runs in the pull request over its lib and bin targets (--all-targets of meerkat-mob took 1226 s on run 38859112297, most of it its 54 test targets and lib test, which lint on the push). A red push: main run shows as a failed CI gate check on the main commit and blocks the release gate (require_ci_green needs a successful exact-main run). The classifier fails closed. Any .rs, Cargo.toml, Cargo.lock, .cargo/, nextest, toolchain, build-wrapper, or ci.yml change yields at least one clippy lane and one unit lane; a Rust path that belongs to no workspace package, a missing diff base, or any global build-configuration path escalates to the whole workspace (packed into at most eight shards by estimated build-and-link cost); a classifier error fails the changes job and therefore the gate. The gate asserts that clippy and unit (or a named unit deferral) ran and succeeded for a build-relevant change, that a push also ran closure-check, the workspace rustdoc, governed JSONL and the confinement lane, and it fails when the classifier produced no lanes. make ci-lanes-selftest (scripts/ci-cargo-lanes-selftest.mjs) pins these properties with fixtures for a core touch, docs-only, Cargo.lock-only, machine-authority-only, an unmapped .rs, and a missing base. scripts/ci-gate-consistency-selftest.mjs, in the same target, runs the gate’s own shell on the planner’s plans for both events, a release projection and a dispatch, with every job’s if: evaluated: the gate must pass when every lane the plan selected ran and passed, and fail when a lane that ran fails or is skipped instead (its allowlist names, with the reason, the lanes a dispatch’s gate does not require). PR CI runs the directly changed packages, not their dependents: measured on a 4-vCPU runner, compiling the reverse-dependency closure of meerkat-core (44 of 50 crates) takes 15-18 minutes even when fanned out, which does not fit the budget together with the local pre-push hook. Dependents are covered on the push by closure-check (compile only) and the closure-tests rows, and by the full workspace lanes in nightly and on the release tag. The gate emits a schema-4 attestation for successful pushes to main and to release/** branches, binding the repository, exact commit SHA, Git tree SHA, workflow run and attempt, event, and ref. Its backend is github-hosted-cargo; plan records the selected mode and counts, and component_results records every lane result. PRs and failed runs emit no such attestation. .github/workflows/cargo.yml remains a separate reusable/manual workflow with its own Cargo lane gate; it is a diagnostic fallback, not the per-push job graph.

Known limits

  • The sccache read-only mode is a client-side setting. A run that only reads the bucket (a pull request from this repository, a dispatch on any other ref) still authenticates as the identity push runs write with: any of its steps can read the credentials file SCCACHE_GCS_KEY_PATH names. Only a separate read-only identity for those runs, an IAM change outside this repository, closes that.
  • The release-branch prefix that lets a dispatch write the bucket is no authorization control either: any writer can create a rel/** branch and dispatch it, as any writer can push to release/**. The write boundary is repository write access.
  • The bearer token in that credentials file (gha-creds-*.json) is the job’s Actions runtime token: the runner sets ACTIONS_ID_TOKEN_REQUEST_TOKEN to the same token. No step exports the token to the lanes’ run steps, but a run step of a lane that authenticated to GCS can still read it there. The token reaches the repository’s Actions caches; no release job restores one, so the reach ends at CI caches.
  • Nightly, cargo.yml and the dense Mob topology workflow still install sccache through mozilla-actions/sccache-action (with a pinned version), by design: they use its GitHub Actions cache backend, which needs the token the action exports. A failed install there leaves the job without sccache too: the step continues on error, and the next one sets an empty RUSTC_WRAPPER with a warning.
  • A CI lane downloads the pinned sccache release with the job’s own token first (gh release download) and from the anonymous release URL only when that fails. A lane that cannot download it either way (every try answered 403, say), or whose sccache server does not start, runs without sccache, with a warning that names the cause: it compiles cold, and a pull request’s gate still holds it to the lane execution budget. A download whose sha256 does not match the pin still fails the lane. Once installed, a server that stops answering mid-job degrades a compile to a plain rustc run (SCCACHE_IGNORE_SERVER_IO_ERROR) instead of failing it.
  • The lanes fetch the mold linker (.github/actions/setup-mold-ci, in place of rui314/setup-mold), cargo-nextest and cargo-deny (setup-cargo-tool-ci, in place of taiki-e/install-action: the releases and sha256 values its manifest named, installed where it installed them), the TLC jar (setup-tlc-ci) and wasm-pack from their GitHub releases the same way: through the GitHub API with the job’s own token first (scripts/ci-release-asset-download; a fork’s pull request gets a read-only token, which is enough), and from the anonymous release URL as before only when there is no token or that fails. The TLC jar, nextest and cargo-deny are checked against their pinned sha256 either way, before anything is unpacked or installed; mold and wasm-pack pin a release, as before, but no checksum. A token sent to the anonymous URL is not enough: on 2026-11-08 that host answered anonymous downloads with 504 for hours, also those an action sent with the job’s token, while the API’s downloads kept working.
  • actions/setup-python and actions/setup-node (for a version the runner image does not cache), bazelbuild/setup-bazelisk and mozilla-actions/sccache-action still download from the anonymous release URL: no input of theirs routes the download through the API.
  • A release job whose checkout can be an older tag runs its setup action from .release-tooling, a sparse checkout of the workflow’s own commit, but a local action that action calls in turn (uses: ./.github/actions/...) resolves against the job’s workspace, the tag’s checkout. No release job reaches one (setup-buildbuddy-ci calls setup-tlc-ci for its full profile only), and the BuildBuddy release validation, which runs the lane actions of the tree it checks out, refuses a release tag.

CI Retry And Concurrency

Pull request CI uses one concurrency group per PR (pr-<number>) with cancel-in-progress: true. Starting another PR-associated attempt in that group automatically cancels the older in-progress attempt, even when nobody issues an explicit cancel command. Pushes to main share one group (main-main-superseded) with cancel-in-progress: true: a newer main commit supersedes the still-running first attempt of an older one, so only the newest main commit is fully tested while merges stream in, and its run covers every superseded change. The account runs at most 40 hosted jobs at once and a main run starts 35-50 of them; in the 2026-10-01 02:27-03:50 UTC merge burst, 46% of the main-run job-minutes (806 of 1760) ran after a newer main commit had been pushed, while pull-request jobs waited up to 11 minutes for a runner. Three kinds of run keep one group per commit (main-main-<sha>) and are never superseded:
  • a release commit, whose head commit message contains chore: release v, because require_ci_green in the release workflow needs a successful exact-main run on exactly the release commit;
  • every re-run attempt (run_attempt above 1): when a superseded commit is needed after all (a tag on it, or bisecting a red main), gh run rerun <id> gives it an attempt that runs to the end and emits its attestation. require_ci_green prints that command when it finds the release commit’s run cancelled;
  • workflow_dispatch runs.
The CI gate job runs under !cancelled(), so a superseded run surfaces as cancelled rather than leaving a failed CI gate context on the head, and it never builds an exact-tree attestation; require_ci_green accepts only a run that completed with success. The reusable BuildBuddy workflow (nightly and release) serializes its shared executor lifecycle with its own concurrency group; that group’s cancel-in-progress is false. Treat check-context precedence as part of retry recovery:
  1. Preserve the reviewed head commit when approvals or downstream evidence are bound to its exact bytes.
  2. Compare the active step and elapsed time with a successful exact-head run before classifying a slow job as hung.
  3. When a new PR-associated attempt is necessary, expect the automatic cancellation. The recovery attempt must complete every selected lane and reach the top-level CI gate.
  4. Inspect every check suite on the reviewed head, including CI gate. Check the repository’s actual required-check settings rather than inferring them from workflow names (CI gate is the only required context today). A green shard or successful manual Cargo workflow is not proof that the active CI graph or a required context in another suite is green.
Do not infer mergeability from a workflow’s overall conclusion alone, and do not apply a creation-time or completion-time recency rule to suites on the same head. Historical Cargo-topology recovery observation: the older graph published GHA Cargo / Cargo lane gate alongside CI gate; that is not the current workflow inventory. In that observed recovery, auto-merge remained blocked after the newest of three suites reported a successful CI gate; it fired only after the last remaining suite reported that context successful. Treat each suite that publishes a required context as independent recovery evidence, because neither the newest nor the latest-completed result was observed to supersede the others. In this recovery, gh pr checks reported pass while guarded merge remained blocked, so an attempted merge that stays blocked is stronger evidence than that flattened view. Inspect the suite records directly when the rollup and merge policy disagree:

Nightly Lanes

Everything pull-request CI deliberately does not run executes daily and on demand (.github/workflows/nightly.yml) on GitHub-hosted runners:
  • the full workspace unit lane (make test-unit) and integration-fast lane (make test-int), covering every crate rather than the changed packages
  • the deterministic e2e-fast lane
  • the 300-member/150-peer dense Mob topology stress (mob-dense-topology.yml)
  • bounded TLC machine verification (make machine-verify)
  • the Python + TypeScript SDK suites, unconditionally
  • clippy across all targets and all features for the whole workspace
  • feature-matrix lint and tests, minimal-feature and surface-modularity tests
  • e2e-system, the full Web SDK and WASM browser contract suites, release packaging dry-runs, the security audit, and the Meta docs mirror lag
  • the whole BuildBuddy/Bazel graph (buildbuddy.yml in full-fresh mode): prebuild, fmt/static, the native nextest lanes, governance (seam-inventory, rmat-audit, machine authority), SDK suites, wasm-check, minimal-feature, feature matrix, and audit

Published crates standalone

Since 0.9.4 every crate the release publishes passes its own tests in its unpacked package. .github/workflows/published-crates-standalone.yml runs daily, after the nightly chains, and on dispatch (gh workflow run published-crates-standalone.yml --ref <branch>). It packages every release crate and unpacks the packages into a synthetic workspace. Per crate, without sibling feature unification, it runs plain cargo test with the feature set the crate’s [package.metadata.meerkat.standalone-tests] publishes (the crate’s README prints the command), checks that cargo test --no-run compiles with the default features, and compares each test target’s test names with the workspace’s, minus the files and tests ci/standalone-workspace-only.toml declares workspace-only. meerkat, meerkat-runtime and meerkat-mob also declare the 8 MiB test thread stack the workspace runs their tests with, and their README command sets RUST_MIN_STACK=8388608; the other crates’ suites have not been measured at Rust’s 2 MiB default and may need the same setting. The fmt-governance job of pull-request CI runs the static half (make published-crates-standalone-selftest and make check-published-crates-standalone-guard). Publication does not wait for the census yet. rkat’s docs-examples test parses crates/meerkat-cli/tests/fixtures/doc_examples.txt, which the package carries, instead of reading docs/. The repository test doc_examples_fixture fails until that file lists exactly the fenced rkat invocations in docs/**/*.mdx: after adding or changing one, refresh it with UPDATE_DOC_EXAMPLES_FIXTURE=1 cargo test -p rkat --test doc_examples_fixture.

Release Validation

The tag release workflow requires successful exact-main CI for the release commit and downloads that run’s schema-4 attestation (legacy schema 1-3 attestations are still accepted for older commits). While the repository variable MEERKAT_GCP_BUILDBUDDY_ENABLED is true it then runs the whole BuildBuddy/Bazel graph in full-fresh mode (release_validate_buildbuddy_full) as the release validation gate; while that self-hosted graph is off (the default) the tag skips the Cargo validation lane (release_validate_cargo) only when the attestation proves that exact-main run ran every lane it repeats (the plan’s generated_contract ratchets and a green sdk-host); otherwise, and for manual dispatches on the GitHub-hosted backend, the tag validates with Cargo. It still builds and publishes the platform-specific release artifacts from the attested source. The critical Linux and macOS arm64 archives publish first; Windows binaries are cross-compiled and packaged with cargo-xwin (clang-cl, lld-link, the Windows SDK) on a hosted Ubuntu runner, then executed for verification on a windows-2025-vs2026 runner, and Intel macOS binaries are cross-compiled on Apple silicon (on the owner’s BuildBuddy backend by its Apple-silicon executors, then run and inspected on a hosted Apple silicon runner; on the GitHub-hosted backend by that runner); those trailing archives are added afterwards, never replacing a published asset. A pre-tag prebuild (make release-prebuild, a release.yml dispatch on main that publishes nothing) can build all of those artifacts for the release commit before the tag; the tag run of that commit, or of a later commit whose changes since touch no input of Core’s release artifacts (scripts/release-prebuild-equivalent: other products’ roots except their Cargo.toml files, product changelogs and a short list of CI-only files), then publishes them after the same attestation checks as an asset recovery instead of rebuilding. A run of exactly the tag commit is preferred, and a release built from an earlier commit’s prebuild carries prebuild-source-<version>.txt, which names both commits. Registry publication remains GitHub-hosted. The repository pre-push dispatcher follows the same identity rule. A complete successful hook run writes a local Git-tree-keyed evidence stamp. A later push of the identical tree, including its release tag, validates the pushed object and checked-out HEAD identity and then reuses the complete gate result. BuildBuddy remains an opt-in developer backend, the nightly full graph, and the release-tag validation and Linux and macOS (arm64 and Intel) packaging backend. .github/workflows/buildbuddy.yml is workflow_call-only; nightly.yml and release.yml call it, ci.yml never does.

Reasoning effort execution receipt

scripts/reasoning-effort-receipt.mjs writes, for one commit, what was executed for the provider-neutral reasoning effort level and what was not: one JSON document (scripts/reasoning-effort-receipt.schema.json, schema version 1) and a Markdown file rendered from it. No workflow runs it or attaches its output to a release yet.
  • The row manifest scripts/reasoning-effort-receipt-rows.json lists the cells (provider family, backend, provider API), the legs (in process, built binary, live) and the rows with the test that covers each. Where a row picks its model or level when it runs, the manifest names the rule.
  • Each leg has one result file, <leg>.json: record writes it from the run’s output (cargo nextest run --message-format libtest-json-plus, which names every test, or the human log, which gives the summary’s counts and the tests whose status lines it shows), or records that the leg did not run and why. A leg result holds counts, test names and typed cells, never the run’s text.
  • The route rows print what they picked: one line per cell, the prefix reasoning-effort-receipt-cell and one JSON object of the row’s name, the cell, the model, the level and the route. record reads those lines and nothing else a test prints: in the JSON output of a run with --no-capture, or in the log of a run with --success-output final. On an executed entry the printed model and level replace the manifest’s rule; a line of a cell the manifest does not list, or one that disagrees with a value the manifest fixes, is an error. The row that refuses levels outside a route’s set refuses several on some routes: it prints one line per refused level (the manifest marks it lines: per_level), and an executed cell of it has one entry per level, row/cell:level, the lowest level first.
  • A second thing a test asserts gets a row of its own. The manifest row names the same test and a case, and the test prints the lines of that case with that word as case; those lines belong to the case’s row and to no other. The refusing route row has one: on Copilot Messages it also refuses a level the route accepts when the session sets a second thinking type (routes.second_thinking_type). The selftest fails when the route rows assert a refusal reason that is the outcome of no manifest row.
  • A row whose cells only its run knows (the models of a live account) prints each cell it reached: the prefix reasoning-effort-receipt-observation and one JSON object that is exactly an observation of the inputs schema (row, provider family, backend, API, model, requested level, outcome, result). record reads those lines from the run’s output the same way and refuses an object with any other member; --observations FILE adds a list from a file. The receipt reads a row’s observations only from a leg result that names the row’s own test with a pass or a fail: beside a test the run skipped or did not run they make no entry, and the row is listed as source only with that reason. A failed cell may carry the HTTP status and an error code as typed fields, the code a word of the row’s own failure classes; never a provider’s text or message. The two live rows are copilot_live_reasoning_effort_matrix and direct_provider_live_reasoning_effort of meerkat-integration-tests, both ignored by default.
  • A live row prints the constant <redacted> for a model id it withheld. The receipt records such a cell with its own result and the model as observed with its id withheld, never as an id. The manifest may name models a live row is to show (named_models: a cell and a model id). A cell shows one only when it names that id on that route; a named model no cell shows is an entry of its own and a gap, model_not_proven when a cell of the route had its id withheld and model_not_observed otherwise. The totals count those two on a line of their own, apart from the rows that did not run.
  • A row may name a test that is written and not yet in the tree, with the mark awaited on the test. A commit that does not define the test then lists the row’s cells as not_run with the reason test_not_at_commit; without the mark a test the commit does not define is an error. The mark goes when the test lands: the selftest fails while a marked test is in the tree. No row of the manifest carries the mark at present: the built-binary rows of the three direct provider routes and of the self-hosted server, which were written ahead of their tests, name tests the tree defines.
  • make reasoning-effort-receipt RESULTS=<directory> OUT=<directory> (or the script’s generate) reads the version, the manifest and the test sources from the commit and writes both files. An entry is executed only when its leg ran at that commit and reported the entry’s own result; it then names the result file and its digest. A row in the source without such a result is source_only, a cell no test covers is not_run, and each is listed again under gaps. A leg without a result file, a result of no leg, a file that does not match its schema, and a row whose test the commit does not define (unless the manifest marks it awaited) are errors.
  • The output is the same bytes for the same inputs, and holds no path, host name, URL, credential or e-mail address: such a string in an input is refused. verify checks a written receipt against its schema, its Markdown file and its result files’ digests.
  • A receipt is written whatever it holds. --check (on generate and verify; CHECK=1 on the make target) judges it and exits 3 on any failed entry and on any gap of the in-process and built-binary legs, which need no credential and can always run. Gaps of the live legs never fail it: it names each with its reason (a named model that is not proven is one of them).
make ci-lanes-selftest runs scripts/reasoning-effort-receipt-selftest.mjs, which also fails when the committed manifest names a test the tree does not define, or when the cell, case and row names the route rows and the live rows print are not the manifest’s.

Doctor

Run:
The doctor checks the API key, pinned bb CLI, generated Bazel files, selector behavior, lane isolation, and docs wiring without printing secrets.

Multi-Agent Work

Separate Git worktrees are already isolated by path hash. If multiple agents share one checkout, set a distinct RUST_LANE_ID per agent when you want stable warm local output roots.

See Also