Local Commands
Use these commands for normal development:scripts/affected-tests.mjs selects the tests a change affects with the CI
planner; without --run it prints the commands instead of running them.
Avoid raw cargo for normal repo work. When you need targeted Cargo commands,
use ./scripts/repo-cargo so output roots and multi-agent work stay isolated.
A crate’s own tests build with its test features: the features its
[package.metadata.meerkat.standalone-tests] lists, plus workspace-tests
when it declares that feature. No dev-dependency turns them on, so a test
command passes them. node scripts/core-workspace-args.mjs --tests -p <crate>
prints a crate’s selection with them, and --tests alone prints Core’s
whole-workspace selection with every member’s. Run tests in that form:
cargo check, clippy and an editor’s
check run are unaffected. The check sits in the crate’s lib test, so a run
narrowed to an integration target (--test <target>) is not stopped: without
the flags it compiles the tests they gate out without a word. Pass them there
too, plus any feature the target’s own #![cfg(feature = ...)] names
(meerkat-core’s wholeblob_compaction_media_guard needs test-support; a
second --features adds it):
rust-analyzer.cargo.features to "all": without
features it treats the code the test features gate as inactive, and its Run
test lens stops on the check. A package-qualified feature list does not work
there, because Cargo refuses a feature of a package the command does not
select and rust-analyzer’s runnables select one package. "all" also turns
on every other feature (the live, realtime and GPT Live ones, and the
products’), so the editor’s check compiles more than a default build does;
it is still the one global value that works.
BuildBuddy Opt-In
SetMEERKAT_BUILDBUDDY=1 to route the same Make lanes through the BuildBuddy
backend:
BUILDBUDDY_DRY_RUN=1 with explicit BuildBuddy Make targets to inspect the
selected command without running it.
The explicit targets are make buildbuddy-build, make buildbuddy-check,
make buildbuddy-clippy, make buildbuddy-test,
make buildbuddy-test-unit, make buildbuddy-test-int,
make buildbuddy-e2e-fast, and make buildbuddy-e2e-system. The live-provider
targets are make buildbuddy-e2e-live and make buildbuddy-e2e-smoke when
credentials are configured.
The Make targets delegate through scripts/run-build-backend-lane; when
BuildBuddy is enabled, that switch calls scripts/buildbuddy-dev.
Test Lanes
The canonical e2e lane taxonomy lives in
tests/integration/src/e2e_lanes.rs. Scripts and BuildBuddy targets route to
that taxonomy rather than inventing separate lane names.
Public Live real-audio smoke
Turbo S scenario 98 remains the existinge2e_smoke_s98_gpt_live_public_playback_settlement_and_reopen selector:
gpt-live-1/OpenAI test, not a mock or a readiness
probe. It requires OPENAI_API_KEY (or RKAT_OPENAI_API_KEY) with Live
voice-session access, Node/npm, Chromium, and working provider WebRTC.
Missing credentials, refused transport, and absent speech fail the selected
test; none becomes a successful skip. The canonical lane installs the
browser prerequisites.
The existing browser peer sends checked-in synthetic WAV speech through a
WebAudio media track, never a microphone or user recording. Every checked
exchange requires new inbound RTP and measured non-silent decoded audio
(at least 100 ms) or non-silent WebAudio analyser windows. RTP energy is
diagnostic only; an absent browser statistic remains absent, never a measured
zero or successful audio evidence.
Signaling, transcript deltas, and provider completion events cannot satisfy
that evidence gate. Numeric GPT_LIVE_PUBLIC_AUDIO records report measured
audio. The scenario verifies exact close/reopen on the same durable session,
with a five-second exact-close ceiling and a 90-second open/reconnect ceiling,
spoken code-word recall after reopening, and real tool execution through the
feature-owned ExistingMember background-delegation policy. Generated
operation custody must identify the original member and report a completed
turn; its canonical history must contain a successful pwd result linked to
the exact shell tool call (zero exit, no timeout, and the actual project
directory), followed by a provider assistant answer. Error-only tool results
cannot satisfy this gate. No disposable fork is spawned. Its ordinary text model
and session identity remain unchanged. S98 calls ServiceMemberLiveHost’s
shared, exact-receipt lifecycle directly; S97 retains the RPC/durable-fork
vertical. After the initial delegation completes, S98 commits a later ordinary
background turn that changes the code word, then requires the unchanged live
channel to speak that new word using fresh measured audio. This is separate
from delivery of the initial delegation’s result.
The existing executor model defaults to gpt-5.6-sol; an operator
may select another available ordinary text model with
GPT_LIVE_E2E_EXECUTOR_MODEL without changing the Live channel’s gpt-live-1.
S98’s Bazel shard remains
//:e2e_smoke_turbo_s_s98; both backends use the same Rust catalog and test.
Scenario 99 is a separate opt-in paid acceptance scenario; S97/S98 retain their
existing selectors and behavior:
//:e2e_smoke_turbo_s_s99; it carries the same
public Live test binary and routes through the canonical Rust catalog.
S99 composes LiveContextSummaryPolicy with Concurrent bootstrap and an
externally gated, content-only summarizer held for at least 20 seconds. The
gate carries permission only: after release, a separate factory-built LLM
client summarizes the immutable source snapshot using its exact model/provider
and auth binding. No canned summary is used in paid acceptance. The caller
bounds this real call to 90 seconds and 1024 output tokens, disables caller
and provider-native tools, and requires measured input/output token usage.
S99 requires an exact-receipt Active owner, connected WebRTC, fresh provider
output transcript honestly lacking an unpredictable historical phrase, and
at least 100 ms of fresh decoded non-silent native voice before releasing
the summary. Analyser-only activity, RTP, silence, or transcript alone cannot
qualify S99. Checked-in synthetic WAVs are the only speech input; a zero-gain
oscillator keeps the media track and RTP alive between speech fixtures.
S99 selects ProviderManagedUnmeasured bookkeeping, with no output queue,
polling, publication receipt pump, or manual playback completion. Its required
binder publisher is a fail-closed guard against unexpected actionable playback
publication; the owner must settle internally without calling it. Provider
transcripts and decoded-audio assertions are evidence, never settlement
signals. S98 retains its separate caller-confirmed playback path.
While preparation remains Generating, an ordinary text update and a newer
spoken correction are admitted, and an existing-member delegation must complete
with the same successful pwd evidence as S98. After receipt-validated
ProviderAcknowledged, fresh marked spoken queries must recall the historical
phrase while retaining both newer facts over the old summary. Historical
thinking-context delivery requests no speech, but releasing its barrier can
legitimately release queued speakable work results. S99 neither suppresses
those results nor asserts full-duplex silence. Exact thinking-append/ACK
correlation and absence of fabricated user turns belong to deterministic
provider/owner tests with attribution. Further
close/reopen cycles attempt to release an obsolete job (or observe its
cancellation) while a replacement job remains gated: the replacement must
still report preparation and honestly lack the history until its own job is
released. All waits are bounded (20-minute inner
scenario, 25-minute catalog command). Missing auth is a failure, not a skip.
Compilation and deterministic catalog/audio-evidence tests are not evidence
that this paid scenario has passed; native-provider qualification requires
explicit execution of the selector above.
The dense Mob topology stress has a dedicated GitHub-hosted Linux workflow,
.github/workflows/mob-dense-topology.yml. Nightly calls it (its archive
build alone takes 13-17 minutes on a hosted runner, so it does not fit the
pull-request budget), and the diagnostic Cargo workflow also runs it from the
existing unit-test archive. The test owns a five-minute wire-materialization budget; its nextest
profile reports the named test every minute and terminates a whole-test stall
after eight minutes. The result is bounded to the hosted runner environment:
local macOS
and BuildBuddy runs are supplementary evidence, not latency predictions or
substitutes for this lane.
Architecture Gates
Machine and surface contracts have direct Make targets:make machine-verify is the bounded TLC lane used
for normal validation. Use make machine-verify-full only when you deliberately
need the unbounded verification set.
CI Shape
Pull-request CI is.github/workflows/ci.yml. It runs on pull requests,
pushes to main and to release/** integration branches, and manual
dispatches (any other branch head runs once, through its pull request), entirely with Cargo on GitHub-hosted ubuntu-24.04 runners
(4 vCPU). A pull request runs what fits the owner’s cap of 20 minutes from
the start of the pre-push hook to a green CI gate, and the gate also
enforces 1080 s of lane execution and a 1200 s push-to-terminal ceiling on
it, runner queue excluded; the push runs every lane.
Compiled
objects are cached by sccache (a release pinned by sha256, content-addressed
and shared by every lane; push and schedule runs, and a dispatch on a
release branch (rel/** or release/**, so it warms the bucket for that
tree’s pull request and main push), set the client to write it, every other
run, a pull request included, to only read it, a client-side
setting, see Known limits);
rust-cache holds only the
registry, because per-lane target caches exceeded the repository’s 10 GB
cache limit in one pull request:
What runs where (pull_request is budgeted; push: main is not; a push to a
release/** integration branch runs the same push-only lanes as main, so
reverse-dependency suites run on every release merge, but only main writes
the exact-tree attestation; nightly and the release tag carry the rest):
Reader rows. Some tests read files that belong to no crate: the rkat
doc-example test walks
docs/, the machine-authority docs alignment test
reads docs/ and .claude/skills/, xtask’s static lanes read
CHANGELOG.md, and so on. scripts/ci-non-rust-readers.json lists every
such input and its reader (scripts/ci-non-rust-readers-guard.mjs keeps it
complete, and fails a reader whose named test its features compile out),
and a change to one adds the exact target that reads it to the
integration lane: -p the reader with the inventory’s features, its
lib, bin, test or example target (the lib and bins for a reader in
library or build-script code: its test targets, like those of a pull
request’s code change, are the push’s closure-tests rows’, which also
keep the nested-Cargo canaries) and a filter on the reading tests, where
no other lane of the plan runs it. A pull request leaves the readers that compile
meerkat-mob (named) and the readers keyed only by the whole checkout (live
audits that every path selects) to the push. A push runs its readers’
test targets in closure-tests, which build each package with the
features a whole-workspace build resolves, and every package’s lib and
bins in main-unit; a test target that needs other features (such as
xtask’s machine-authority contracts or the gpt-live-replay suite) gets
a reader row with them, and a push with no main-unit (no Rust-relevant
path, such as a docs-only push) gets reader rows for the lib, bin and
example targets. A reader whose features need live-provider credentials
(integration-real-tests, openai-live-e2e) runs in no CI lane: the plan
names it (reader_not_run) and the CI gate prints it. Documentation inside a
crate is no build input unless the crate compiles it in
(include_str!/include_bytes! in library, binary or build-script code,
such as the crates’ skills/*/SKILL.md); that selects the crate like a
change to its code.
Long rows first. clippy, main-unit, closure-tests and feature-check
run in two waves, each wave a job of its own (<lane> and <lane>-rest).
The rows that took 15 minutes or more on a green run, and rows never
measured, start right after changes, as do the closure check’s
per-product jobs; the shorter rows, and the short lanes (ratchets,
integration, governed-jsonl, sdk-host, example-web, wasm-timers,
bazel-graph), wait for the stagger job, which sleeps 30 seconds on a
push and not at all on a pull request. The account’s hosted runners take
about 40 jobs at a time and pick queued jobs in no reliable order, so on a
whole-workspace push of about 170 jobs a long row picked late set the wall
time. The measurements are
scripts/ci-row-durations.json, regenerated from green runs with
scripts/ci-row-durations.mjs --repository OWNER/REPO --output scripts/ci-row-durations.json RUN_ID...; a stale entry only moves a row
between waves. A row is keyed by its name without the member count of a
packed row (core+12 is core, openai+8 bazel set is openai bazel set),
so a merge that moves line counts and repacks the rows keeps a row’s
measurement while the bin’s heaviest member, which names it, stays the
same. The CI gate counts a two-wave lane as run when every wave
that holds rows succeeded. Other runs on the account (pull-request CI,
nightly, dispatches) take runners from the same pool: keep them off while a
release commit’s main run is in progress.
“Deferred” lanes run on the push of the merge to main (or a release/**
branch), whose CI must be green before a release tag. A pull request’s plan
names them (deferred_to_push) and the gate prints them. They are deferred
because each needs 9-30 minutes on a 4-vCPU runner, mostly to compile the
meerkat-mob chain (meerkat-mob alone compiles for 5-11 minutes whenever it
misses the cache), and the owner’s cap is 20 minutes from the start of the
pre-push hook to a green CI gate on every pull request.
How the budget is measured: the gate reads every completed lane from the
Actions jobs API and splits its time into runner queue (created_at to
started_at) and execution (started_at to completed_at). On pull requests
it enforces two limits:
- Lane execution, 1080 s (
CI_MAX_EXEC_SECONDS): each lane’s own run time plus the run time of the change classification it waits for (and of thestaggerjob, for a lane that waits for it), over the slowest lane. This is the part the code controls: a lane that grows or an aggregate that drifts fails here. Run times belong to the job, so re-running only the gate cannot turn an overrun green. - Push-to-terminal excluding runner queue, 1200 s
(
CI_MAX_TERMINAL_SECONDS): the runaway backstop on the latest lane terminal minus the runner queue on that lane’s path (its own queue plus the change classification’s, and thestaggerjob’s for a lane that waits for it, in the same attempt; the jobs API creates a job when its needs complete, so a lane created after thestaggerjob completed waited for it). Each lane is timed from the start of the run attempt it ran in: a re-run lane gets a fresh clock, and a carried-over lane keeps its attempt’s clock. Past the ceiling a run’s jobs are stuck. Waiting for runners alone never fails it.
governed-jsonl, mcp-confinement-positive) that the
plan runs only because the change touches its inputs or changes its
definition (GUARD_LANE_INPUTS, GUARD_LANE_DEFINITIONS; the plan exports
the reason as guard_lane_reasons) is outside both: the confinement lane’s
cold build alone takes 27.5-32.5 minutes. The gate still requires it to
succeed, its own timeout-minutes (40 on a pull request) bounds it, and the
gate names it, with its seconds and the reason, in a Guard lane outside the budget warning and the guard_lanes_outside_budget output. A guard lane run
on every Rust change (a push, a dispatch, a release projection) or for an
unknown diff stays in the budget.
Runner queue wait is reported, not budgeted: at 300 s or more on the critical
path (CI_QUEUE_WARN_SECONDS) the gate emits a CI runner queue wait
warning. The account runs at most 40 hosted jobs at once, shared by every
pull request, main push and nightly, and a busy hour fills that cap. The
gate writes a typed verdict (within_budget, queue_delayed,
exec_over_budget, runaway) with the execution path, terminal and queue
seconds to its step outputs and summary, and prints the lane table on every
run that was not cancelled, including runs whose lanes failed, pushes and
dispatches. The budget keeps pull-request feedback fast; a push or a
dispatch is a full validation run, where nothing is enforced and a failure to
measure is only a warning.
Sizing: the owner’s cap is 20 minutes from the start of the pre-push hook to
a green CI gate, and the hook takes about a minute, so 1080 s (18 minutes)
of execution leaves about a minute of slack and 1200 s (20 minutes) of
terminal is the runaway backstop at the cap. On the workspace-mode
pull-request runs 38859112297 and 38931641220 the lanes a pull request still
plans ran at most 854 s (wasm-check; example-web 842 s cold) after 28-31 s
of classification. The earlier budgets, 1500 s and 2700 s, were sized on 66
attempts of 2026-09-30 to 2026-10-01 (execution path max 1244 s) before the
meerkat-mob chain’s and the closure’s lanes moved to the push.
Why the meerkat-mob chain is deferred: measured on 4-vCPU hosted runners,
every unit lane that ever exceeded the budget compiled meerkat-mob’s 442k
lines, either as its own lane (16 min compile plus 4.5 min of tests) or as a
dependent crate rebuilding mob under its own feature unification (rkat, rpc:
17 min). The chain is computed from cargo metadata, and the classifier’s cost
model (1 + (lines + 8000 x dependency closure) / 35000 minutes, with a
20-minute surrogate packing cap per pull-request unit lane; the cap is a
model threshold, not a measured timing) is asserted by the self-test for every
other crate. Clippy of a changed mob-chain crate still runs in the pull
request over its lib and bin targets (--all-targets of meerkat-mob took
1226 s on run 38859112297, most of it its 54 test targets and lib test,
which lint on the push). A red push: main run shows as a
failed CI gate check on the main commit and blocks the release gate
(require_ci_green needs a successful exact-main run).
The classifier fails closed. Any .rs, Cargo.toml, Cargo.lock,
.cargo/, nextest, toolchain, build-wrapper, or ci.yml change yields at
least one clippy lane and one unit lane; a Rust path that belongs to no
workspace package, a missing diff base, or any global build-configuration
path escalates to the whole workspace (packed into at most eight shards by
estimated build-and-link cost); a
classifier error fails the changes job and therefore the gate. The gate
asserts that clippy and unit (or a named unit deferral) ran and succeeded for
a build-relevant change, that a push also ran closure-check, the workspace
rustdoc, governed JSONL and the confinement lane, and it fails when the
classifier produced no lanes.
make ci-lanes-selftest (scripts/ci-cargo-lanes-selftest.mjs) pins these
properties with fixtures for a core touch, docs-only, Cargo.lock-only,
machine-authority-only, an unmapped .rs, and a missing base.
scripts/ci-gate-consistency-selftest.mjs, in the same target, runs the
gate’s own shell on the planner’s plans for both events, a release
projection and a dispatch, with every job’s if: evaluated: the gate must
pass when every lane the plan selected ran and passed, and fail when a lane
that ran fails or is skipped instead (its allowlist names, with the reason,
the lanes a dispatch’s gate does not require).
PR CI runs the directly changed packages, not their dependents: measured on
a 4-vCPU runner, compiling the reverse-dependency closure of meerkat-core
(44 of 50 crates) takes 15-18 minutes even when fanned out, which does not
fit the budget together with the local pre-push hook. Dependents are covered
on the push by closure-check (compile only) and the closure-tests rows,
and by the full workspace lanes in nightly and on the release tag.
The gate emits a schema-4 attestation for successful pushes to main and to
release/** branches, binding
the repository, exact commit SHA, Git tree SHA, workflow run and attempt,
event, and ref. Its backend is github-hosted-cargo; plan records the
selected mode and counts, and component_results records every lane result.
PRs and failed runs emit no such attestation.
.github/workflows/cargo.yml remains a separate reusable/manual workflow with
its own Cargo lane gate; it is a diagnostic fallback, not the per-push job
graph.
Known limits
- The sccache read-only mode is a client-side setting. A run that only
reads the bucket (a pull request from this repository, a dispatch on any
other ref) still
authenticates as the identity push runs write with: any of its steps can
read the credentials file
SCCACHE_GCS_KEY_PATHnames. Only a separate read-only identity for those runs, an IAM change outside this repository, closes that. - The release-branch prefix that lets a dispatch write the bucket is no
authorization control either: any writer can create a
rel/**branch and dispatch it, as any writer can push torelease/**. The write boundary is repository write access. - The bearer token in that credentials file (
gha-creds-*.json) is the job’s Actions runtime token: the runner setsACTIONS_ID_TOKEN_REQUEST_TOKENto the same token. No step exports the token to the lanes’ run steps, but a run step of a lane that authenticated to GCS can still read it there. The token reaches the repository’s Actions caches; no release job restores one, so the reach ends at CI caches. - Nightly,
cargo.ymland the dense Mob topology workflow still install sccache throughmozilla-actions/sccache-action(with a pinnedversion), by design: they use its GitHub Actions cache backend, which needs the token the action exports. A failed install there leaves the job without sccache too: the step continues on error, and the next one sets an emptyRUSTC_WRAPPERwith a warning. - A CI lane downloads the pinned sccache release with the job’s own token
first (
gh release download) and from the anonymous release URL only when that fails. A lane that cannot download it either way (every try answered 403, say), or whose sccache server does not start, runs without sccache, with a warning that names the cause: it compiles cold, and a pull request’s gate still holds it to the lane execution budget. A download whose sha256 does not match the pin still fails the lane. Once installed, a server that stops answering mid-job degrades a compile to a plain rustc run (SCCACHE_IGNORE_SERVER_IO_ERROR) instead of failing it. - The lanes fetch the mold linker (
.github/actions/setup-mold-ci, in place ofrui314/setup-mold), cargo-nextest and cargo-deny (setup-cargo-tool-ci, in place oftaiki-e/install-action: the releases and sha256 values its manifest named, installed where it installed them), the TLC jar (setup-tlc-ci) and wasm-pack from their GitHub releases the same way: through the GitHub API with the job’s own token first (scripts/ci-release-asset-download; a fork’s pull request gets a read-only token, which is enough), and from the anonymous release URL as before only when there is no token or that fails. The TLC jar, nextest and cargo-deny are checked against their pinned sha256 either way, before anything is unpacked or installed; mold and wasm-pack pin a release, as before, but no checksum. A token sent to the anonymous URL is not enough: on 2026-11-08 that host answered anonymous downloads with 504 for hours, also those an action sent with the job’s token, while the API’s downloads kept working. actions/setup-pythonandactions/setup-node(for a version the runner image does not cache),bazelbuild/setup-bazeliskandmozilla-actions/sccache-actionstill download from the anonymous release URL: no input of theirs routes the download through the API.- A release job whose checkout can be an older tag runs its setup action
from
.release-tooling, a sparse checkout of the workflow’s own commit, but a local action that action calls in turn (uses: ./.github/actions/...) resolves against the job’s workspace, the tag’s checkout. No release job reaches one (setup-buildbuddy-cicallssetup-tlc-cifor itsfullprofile only), and the BuildBuddy release validation, which runs the lane actions of the tree it checks out, refuses a release tag.
CI Retry And Concurrency
Pull request CI uses one concurrency group per PR (pr-<number>) with
cancel-in-progress: true. Starting another PR-associated attempt in that
group automatically cancels the older in-progress attempt, even when nobody
issues an explicit cancel command.
Pushes to main share one group (main-main-superseded) with
cancel-in-progress: true: a newer main commit supersedes the still-running
first attempt of an older one, so only the newest main commit is fully tested
while merges stream in, and its run covers every superseded change. The
account runs at most 40 hosted jobs at once and a main run starts 35-50 of
them; in the 2026-10-01 02:27-03:50 UTC merge burst, 46% of the main-run
job-minutes (806 of 1760) ran after a newer main commit had been pushed, while
pull-request jobs waited up to 11 minutes for a runner. Three kinds of run keep
one group per commit (main-main-<sha>) and are never superseded:
- a release commit, whose head commit message contains
chore: release v, becauserequire_ci_greenin the release workflow needs a successful exact-main run on exactly the release commit; - every re-run attempt (
run_attemptabove 1): when a superseded commit is needed after all (a tag on it, or bisecting a red main),gh run rerun <id>gives it an attempt that runs to the end and emits its attestation.require_ci_greenprints that command when it finds the release commit’s run cancelled; workflow_dispatchruns.
CI gate job runs under !cancelled(), so a superseded run surfaces as
cancelled rather than leaving a failed CI gate context on the head, and it
never builds an exact-tree attestation; require_ci_green accepts only a run
that completed with success. The
reusable BuildBuddy workflow (nightly and release) serializes its shared
executor lifecycle with its own concurrency group; that group’s
cancel-in-progress is false.
Treat check-context precedence as part of retry recovery:
- Preserve the reviewed head commit when approvals or downstream evidence are bound to its exact bytes.
- Compare the active step and elapsed time with a successful exact-head run before classifying a slow job as hung.
- When a new PR-associated attempt is necessary, expect the automatic
cancellation. The recovery attempt must complete every selected lane and
reach the top-level
CI gate. - Inspect every check suite on the reviewed head, including
CI gate. Check the repository’s actual required-check settings rather than inferring them from workflow names (CI gateis the only required context today). A green shard or successful manual Cargo workflow is not proof that the active CI graph or a required context in another suite is green.
GHA Cargo / Cargo lane gate alongside CI gate; that is not the current
workflow inventory. In that observed recovery, auto-merge remained blocked
after the newest of three suites reported a successful CI gate; it fired
only after the last remaining suite reported that context successful. Treat
each suite that publishes a required context as independent recovery evidence, because neither
the newest nor the latest-completed result was observed to supersede the
others. In this recovery, gh pr checks reported pass while guarded merge
remained blocked, so an attempted merge that stays blocked is stronger evidence
than that flattened view.
Inspect the suite records directly when the rollup and merge policy disagree:
Nightly Lanes
Everything pull-request CI deliberately does not run executes daily and on demand (.github/workflows/nightly.yml) on GitHub-hosted runners:
- the full workspace unit lane (
make test-unit) and integration-fast lane (make test-int), covering every crate rather than the changed packages - the deterministic
e2e-fastlane - the 300-member/150-peer dense Mob topology stress
(
mob-dense-topology.yml) - bounded TLC machine verification (
make machine-verify) - the Python + TypeScript SDK suites, unconditionally
- clippy across all targets and all features for the whole workspace
- feature-matrix lint and tests, minimal-feature and surface-modularity tests
e2e-system, the full Web SDK and WASM browser contract suites, release packaging dry-runs, the security audit, and the Meta docs mirror lag- the whole BuildBuddy/Bazel graph (
buildbuddy.ymlinfull-freshmode): prebuild, fmt/static, the native nextest lanes, governance (seam-inventory, rmat-audit, machine authority), SDK suites, wasm-check, minimal-feature, feature matrix, and audit
Published crates standalone
Since 0.9.4 every crate the release publishes passes its own tests in its unpacked package..github/workflows/published-crates-standalone.yml runs
daily, after the nightly chains, and on dispatch
(gh workflow run published-crates-standalone.yml --ref <branch>). It
packages every release crate and unpacks the packages into a synthetic
workspace. Per crate, without sibling feature unification, it runs plain
cargo test with the feature set the crate’s
[package.metadata.meerkat.standalone-tests] publishes (the crate’s README
prints the command), checks that cargo test --no-run compiles with the
default features, and compares each test target’s test names with the
workspace’s, minus the files and tests ci/standalone-workspace-only.toml
declares workspace-only. meerkat, meerkat-runtime and meerkat-mob also
declare the 8 MiB test thread stack the workspace runs their tests with, and
their README command sets RUST_MIN_STACK=8388608; the other crates’ suites
have not been measured at Rust’s 2 MiB default and may need the same setting.
The fmt-governance job of pull-request CI runs the static half
(make published-crates-standalone-selftest and
make check-published-crates-standalone-guard). Publication does not wait for
the census yet.
rkat’s docs-examples test parses
crates/meerkat-cli/tests/fixtures/doc_examples.txt, which the package
carries, instead of reading docs/. The repository test
doc_examples_fixture fails until that file lists exactly the fenced rkat
invocations in docs/**/*.mdx: after adding or changing one, refresh it with
UPDATE_DOC_EXAMPLES_FIXTURE=1 cargo test -p rkat --test doc_examples_fixture.
Release Validation
The tag release workflow requires successful exact-main CI for the release
commit and downloads that run’s schema-4 attestation (legacy schema 1-3
attestations are still accepted for older commits). While the repository
variable MEERKAT_GCP_BUILDBUDDY_ENABLED is true it then runs the whole
BuildBuddy/Bazel graph in full-fresh mode (release_validate_buildbuddy_full)
as the release validation gate; while that self-hosted graph is off (the
default) the tag skips the Cargo validation lane (release_validate_cargo)
only when the attestation proves that exact-main run ran every lane it
repeats (the plan’s generated_contract ratchets and a green sdk-host);
otherwise, and for manual dispatches on the GitHub-hosted backend, the tag
validates with Cargo. It still builds
and publishes the platform-specific release artifacts from the attested
source. The critical Linux and macOS arm64 archives publish first; Windows
binaries are cross-compiled and packaged with cargo-xwin (clang-cl, lld-link,
the Windows SDK) on a hosted Ubuntu runner, then executed for verification on
a windows-2025-vs2026 runner, and Intel macOS binaries are cross-compiled on
Apple silicon (on the owner’s BuildBuddy backend by its Apple-silicon
executors, then run and inspected on a hosted Apple silicon runner; on the
GitHub-hosted backend by that runner); those trailing archives are added afterwards, never
replacing a published asset. A pre-tag prebuild
(make release-prebuild, a release.yml dispatch on main that publishes
nothing) can build all of those artifacts for the release commit before the
tag; the tag run of that commit, or of a later commit whose changes since
touch no input of Core’s release artifacts
(scripts/release-prebuild-equivalent: other products’ roots except their
Cargo.toml files, product changelogs and a short list of CI-only files),
then publishes them after the same attestation checks as an asset recovery
instead of rebuilding. A run of exactly the tag commit is preferred, and a
release built from an earlier commit’s prebuild carries
prebuild-source-<version>.txt, which names both commits. Registry
publication remains GitHub-hosted.
The repository pre-push dispatcher follows the same identity rule. A complete
successful hook run writes a local Git-tree-keyed evidence stamp. A later push
of the identical tree, including its release tag, validates the pushed object
and checked-out HEAD identity and then reuses the complete gate result.
BuildBuddy remains an opt-in developer backend, the nightly full graph, and
the release-tag validation and Linux and macOS (arm64 and Intel) packaging
backend.
.github/workflows/buildbuddy.yml is workflow_call-only; nightly.yml and
release.yml call it, ci.yml never does.
Reasoning effort execution receipt
scripts/reasoning-effort-receipt.mjs writes, for one commit, what was
executed for the provider-neutral reasoning effort level and what was not:
one JSON document (scripts/reasoning-effort-receipt.schema.json, schema
version 1) and a Markdown file rendered from it. No workflow runs it or
attaches its output to a release yet.
- The row manifest
scripts/reasoning-effort-receipt-rows.jsonlists the cells (provider family, backend, provider API), the legs (in process, built binary, live) and the rows with the test that covers each. Where a row picks its model or level when it runs, the manifest names the rule. - Each leg has one result file,
<leg>.json:recordwrites it from the run’s output (cargo nextest run --message-format libtest-json-plus, which names every test, or the human log, which gives the summary’s counts and the tests whose status lines it shows), or records that the leg did not run and why. A leg result holds counts, test names and typed cells, never the run’s text. - The route rows print what they picked: one line per cell, the prefix
reasoning-effort-receipt-celland one JSON object of the row’s name, the cell, the model, the level and the route.recordreads those lines and nothing else a test prints: in the JSON output of a run with--no-capture, or in the log of a run with--success-output final. On an executed entry the printed model and level replace the manifest’s rule; a line of a cell the manifest does not list, or one that disagrees with a value the manifest fixes, is an error. The row that refuses levels outside a route’s set refuses several on some routes: it prints one line per refused level (the manifest marks itlines: per_level), and an executed cell of it has one entry per level,row/cell:level, the lowest level first. - A second thing a test asserts gets a row of its own. The manifest row names
the same test and a
case, and the test prints the lines of that case with that word ascase; those lines belong to the case’s row and to no other. The refusing route row has one: on Copilot Messages it also refuses a level the route accepts when the session sets a second thinking type (routes.second_thinking_type). The selftest fails when the route rows assert a refusal reason that is the outcome of no manifest row. - A row whose cells only its run knows (the models of a live account)
prints each cell it reached: the prefix
reasoning-effort-receipt-observationand one JSON object that is exactly anobservationof the inputs schema (row, provider family, backend, API, model, requested level, outcome, result).recordreads those lines from the run’s output the same way and refuses an object with any other member;--observations FILEadds a list from a file. The receipt reads a row’s observations only from a leg result that names the row’s own test with a pass or a fail: beside a test the run skipped or did not run they make no entry, and the row is listed as source only with that reason. A failed cell may carry the HTTP status and an error code as typed fields, the code a word of the row’s own failure classes; never a provider’s text or message. The two live rows arecopilot_live_reasoning_effort_matrixanddirect_provider_live_reasoning_effortofmeerkat-integration-tests, both ignored by default. - A live row prints the constant
<redacted>for a model id it withheld. The receipt records such a cell with its own result and the model as observed with its id withheld, never as an id. The manifest may name models a live row is to show (named_models: a cell and a model id). A cell shows one only when it names that id on that route; a named model no cell shows is an entry of its own and a gap,model_not_provenwhen a cell of the route had its id withheld andmodel_not_observedotherwise. The totals count those two on a line of their own, apart from the rows that did not run. - A row may name a test that is written and not yet in the tree, with the
mark
awaitedon the test. A commit that does not define the test then lists the row’s cells asnot_runwith the reasontest_not_at_commit; without the mark a test the commit does not define is an error. The mark goes when the test lands: the selftest fails while a marked test is in the tree. No row of the manifest carries the mark at present: the built-binary rows of the three direct provider routes and of the self-hosted server, which were written ahead of their tests, name tests the tree defines. make reasoning-effort-receipt RESULTS=<directory> OUT=<directory>(or the script’sgenerate) reads the version, the manifest and the test sources from the commit and writes both files. An entry isexecutedonly when its leg ran at that commit and reported the entry’s own result; it then names the result file and its digest. A row in the source without such a result issource_only, a cell no test covers isnot_run, and each is listed again undergaps. A leg without a result file, a result of no leg, a file that does not match its schema, and a row whose test the commit does not define (unless the manifest marks itawaited) are errors.- The output is the same bytes for the same inputs, and holds no path, host
name, URL, credential or e-mail address: such a string in an input is
refused.
verifychecks a written receipt against its schema, its Markdown file and its result files’ digests. - A receipt is written whatever it holds.
--check(ongenerateandverify;CHECK=1on the make target) judges it and exits 3 on any failed entry and on any gap of the in-process and built-binary legs, which need no credential and can always run. Gaps of the live legs never fail it: it names each with its reason (a named model that is not proven is one of them).
make ci-lanes-selftest runs scripts/reasoning-effort-receipt-selftest.mjs,
which also fails when the committed manifest names a test the tree does not
define, or when the cell, case and row names the route rows and the live rows
print are not the manifest’s.
Doctor
Run:bb CLI, generated Bazel files, selector
behavior, lane isolation, and docs wiring without printing secrets.
Multi-Agent Work
Separate Git worktrees are already isolated by path hash. If multiple agents share one checkout, set a distinctRUST_LANE_ID per agent when you want stable
warm local output roots.
