Skip to main content
Make is the developer-facing command surface. Cargo is the default backend. Pull-request CI is Cargo-only on GitHub-hosted runners and selects its lanes from the changed paths. BuildBuddy/Bazel is an opt-in acceleration backend for developers with BuildBuddy access, and the whole BuildBuddy graph runs nightly and as the release-tag validation gate; it never runs in pull-request CI. Local Cargo-default behavior does not select the CI backend.

​
Local Commands

Use these commands for normal development:
Avoid raw cargo for normal repo work. When you need targeted Cargo commands, use ./scripts/repo-cargo so output roots and multi-agent work stay isolated.

​
BuildBuddy Opt-In

Set MEERKAT_BUILDBUDDY=1 to route the same Make lanes through the BuildBuddy backend:
Use BUILDBUDDY_DRY_RUN=1 with explicit BuildBuddy Make targets to inspect the selected command without running it. The explicit targets are make buildbuddy-build, make buildbuddy-check, make buildbuddy-clippy, make buildbuddy-test, make buildbuddy-test-unit, make buildbuddy-test-int, make buildbuddy-e2e-fast, and make buildbuddy-e2e-system. The live-provider targets are make buildbuddy-e2e-live and make buildbuddy-e2e-smoke when credentials are configured. The Make targets delegate through scripts/run-build-backend-lane; when BuildBuddy is enabled, that switch calls scripts/buildbuddy-dev.

​
Test Lanes

The canonical e2e lane taxonomy lives in tests/integration/src/e2e_lanes.rs. Scripts and BuildBuddy targets route to that taxonomy rather than inventing separate lane names.

​
Public Live real-audio smoke

Turbo S scenario 98 remains the existing e2e_smoke_s98_gpt_live_public_playback_settlement_and_reopen selector:
This is a paid public gpt-live-1/OpenAI test, not a mock or a readiness probe. It requires OPENAI_API_KEY (or RKAT_OPENAI_API_KEY) with Live voice-session access, Node/npm, Chromium, and working provider WebRTC. Missing credentials, refused transport, and absent speech fail the selected test; none becomes a successful skip. The canonical lane installs the browser prerequisites. The existing browser peer sends checked-in synthetic WAV speech through a WebAudio media track, never a microphone or user recording. Every checked exchange requires new inbound RTP and measured non-silent decoded audio (at least 100 ms) or non-silent WebAudio analyser windows. RTP energy is diagnostic only; an absent browser statistic remains absent, never a measured zero or successful audio evidence. Signaling, transcript deltas, and provider completion events cannot satisfy that evidence gate. Numeric GPT_LIVE_PUBLIC_AUDIO records report measured audio. The scenario verifies exact close/reopen on the same durable session, with a five-second exact-close ceiling and a 90-second open/reconnect ceiling, spoken code-word recall after reopening, and real tool execution through the feature-owned ExistingMember background-delegation policy. Generated operation custody must identify the original member and report a completed turn; its canonical history must contain a successful pwd result linked to the exact shell tool call (zero exit, no timeout, and the actual project directory), followed by a provider assistant answer. Error-only tool results cannot satisfy this gate. No disposable fork is spawned. Its ordinary text model and session identity remain unchanged. S98 calls ServiceMemberLiveHost’s shared, exact-receipt lifecycle directly; S97 retains the RPC/durable-fork vertical. After the initial delegation completes, S98 commits a later ordinary background turn that changes the code word, then requires the unchanged live channel to speak that new word using fresh measured audio. This is separate from delivery of the initial delegation’s result. The existing executor model defaults to gpt-5.6-sol; an operator may select another available ordinary text model with GPT_LIVE_E2E_EXECUTOR_MODEL without changing the Live channel’s gpt-live-1. S98’s Bazel shard remains //:e2e_smoke_turbo_s_s98; both backends use the same Rust catalog and test. Scenario 99 is a separate opt-in paid acceptance scenario; S97/S98 retain their existing selectors and behavior:
The generated Bazel shard is //:e2e_smoke_turbo_s_s99; it carries the same public Live test binary and routes through the canonical Rust catalog. S99 composes LiveContextSummaryPolicy with Concurrent bootstrap and an externally gated, content-only summarizer held for at least 20 seconds. The gate carries permission only: after release, a separate factory-built LLM client summarizes the immutable source snapshot using its exact model/provider and auth binding. No canned summary is used in paid acceptance. The caller bounds this real call to 90 seconds and 1024 output tokens, disables caller and provider-native tools, and requires measured input/output token usage. S99 requires an exact-receipt Active owner, connected WebRTC, fresh provider output transcript honestly lacking an unpredictable historical phrase, and at least 100 ms of fresh decoded non-silent native voice before releasing the summary. Analyser-only activity, RTP, silence, or transcript alone cannot qualify S99. Checked-in synthetic WAVs are the only speech input; a zero-gain oscillator keeps the media track and RTP alive between speech fixtures. S99 selects ProviderManagedUnmeasured bookkeeping, with no output queue, polling, publication receipt pump, or manual playback completion. Its required binder publisher is a fail-closed guard against unexpected actionable playback publication; the owner must settle internally without calling it. Provider transcripts and decoded-audio assertions are evidence, never settlement signals. S98 retains its separate caller-confirmed playback path. While preparation remains Generating, an ordinary text update and a newer spoken correction are admitted, and an existing-member delegation must complete with the same successful pwd evidence as S98. After receipt-validated ProviderAcknowledged, fresh marked spoken queries must recall the historical phrase while retaining both newer facts over the old summary. Historical thinking-context delivery requests no speech, but releasing its barrier can legitimately release queued speakable work results. S99 neither suppresses those results nor asserts full-duplex silence. Exact thinking-append/ACK correlation and absence of fabricated user turns belong to deterministic provider/owner tests with attribution. Further close/reopen cycles attempt to release an obsolete job (or observe its cancellation) while a replacement job remains gated: the replacement must still report preparation and honestly lack the history until its own job is released. All waits are bounded (20-minute inner scenario, 25-minute catalog command). Missing auth is a failure, not a skip. Compilation and deterministic catalog/audio-evidence tests are not evidence that this paid scenario has passed; native-provider qualification requires explicit execution of the selector above. The dense Mob topology stress has a dedicated GitHub-hosted Linux workflow, .github/workflows/mob-dense-topology.yml. Nightly calls it (its archive build alone takes 13-17 minutes on a hosted runner, so it does not fit the pull-request budget), and the diagnostic Cargo workflow also runs it from the existing unit-test archive. The test owns a five-minute wire-materialization budget; its nextest profile reports the named test every minute and terminates a whole-test stall after eight minutes. The result is bounded to the hosted runner environment: local macOS and BuildBuddy runs are supplementary evidence, not latency predictions or substitutes for this lane.

​
Architecture Gates

Machine and surface contracts have direct Make targets:
These gates protect generated schemas, SDK wrappers, machine artifacts, and runtime authority invariants. make machine-verify is the bounded TLC lane used for normal validation. Use make machine-verify-full only when you deliberately need the unbounded verification set.

​
CI Shape

Pull-request CI is .github/workflows/ci.yml. It runs on pull requests, pushes to main, and manual dispatches (a branch head runs once, through its pull request), entirely with Cargo on GitHub-hosted ubuntu-latest runners (4 vCPU), and it is sized to a 20-minute push-to-terminal budget. Compiled objects are cached by sccache (content-addressed, shared by every lane and readable from main by every pull request); rust-cache holds only the registry, because per-lane target caches exceeded the repository’s 10 GB cache limit in one pull request: What runs where (pull_request is budgeted; push: main is not; nightly and the release tag carry the rest): Why the meerkat-mob chain is deferred: measured on 4-vCPU hosted runners, every unit lane that ever exceeded the budget compiled meerkat-mob’s 442k lines, either as its own lane (16 min compile plus 4.5 min of tests) or as a dependent crate rebuilding mob under its own feature unification (rkat, rpc: 17 min). The chain is computed from cargo metadata, and the classifier’s cost model (1 + (lines + 8000 x dependency closure) / 35000 minutes, 16-minute ceiling per pull-request unit lane) is asserted by the self-test for every other crate. Clippy of a changed mob-chain crate still runs in the pull request (12 min measured for mob itself). A red push: main run shows as a failed CI gate check on the main commit and blocks the release gate (require_ci_green needs a successful exact-main run). The classifier fails closed. Any .rs, Cargo.toml, Cargo.lock, .cargo/, nextest, toolchain, build-wrapper, or ci.yml change yields at least one clippy lane and one unit lane; a Rust path that belongs to no workspace package, a missing diff base, or any global build-configuration path escalates to the whole workspace (packed into at most eight shards by estimated build-and-link cost); a classifier error fails the changes job and therefore the gate. The gate asserts that clippy, unit, and closure-check all ran and succeeded for a build-relevant change, and it fails when the classifier produced no lanes. make ci-lanes-selftest (scripts/ci-cargo-lanes-selftest.mjs) pins these properties with fixtures for a core touch, docs-only, Cargo.lock-only, machine-authority-only, an unmapped .rs, and a missing base. PR CI runs the directly changed packages, not their dependents: measured on a 4-vCPU runner, compiling the reverse-dependency closure of meerkat-core (44 of 50 crates) takes 15-18 minutes even when fanned out, which does not fit the budget together with the local pre-push hook. Dependents are covered by closure-check (compile only) in PR CI and by the full workspace lanes in nightly and on the release tag. The gate emits a schema-4 attestation for successful main pushes, binding the repository, exact commit SHA, Git tree SHA, workflow run and attempt, event, and ref. Its backend is github-hosted-cargo; plan records the selected mode and counts, and component_results records every lane result. PRs and failed runs emit no such attestation. .github/workflows/cargo.yml remains a separate reusable/manual workflow with its own Cargo lane gate; it is a diagnostic fallback, not the per-push job graph.

​
CI Retry And Concurrency

Pull request CI uses one concurrency group per PR (pr-<number>) with cancel-in-progress: true. Starting another PR-associated attempt in that group automatically cancels the older in-progress attempt, even when nobody issues an explicit cancel command. Pushes to main use one group per commit (main-main-<sha>) with no cancellation, so back-to-back merges each complete their run and emit their attestation. The CI gate job runs under !cancelled(), so a superseded run surfaces as cancelled rather than leaving a failed CI gate context on the head. The reusable BuildBuddy workflow (nightly and release) serializes its shared executor lifecycle with its own concurrency group; that group’s cancel-in-progress is false. Treat check-context precedence as part of retry recovery:
  1. Preserve the reviewed head commit when approvals or downstream evidence are bound to its exact bytes.
  2. Compare the active step and elapsed time with a successful exact-head run before classifying a slow job as hung.
  3. When a new PR-associated attempt is necessary, expect the automatic cancellation. The recovery attempt must complete every selected lane and reach the top-level CI gate.
  4. Inspect every check suite on the reviewed head, including CI gate. Check the repository’s actual required-check settings rather than inferring them from workflow names (CI gate is the only required context today). A green shard or successful manual Cargo workflow is not proof that the active CI graph or a required context in another suite is green.
Do not infer mergeability from a workflow’s overall conclusion alone, and do not apply a creation-time or completion-time recency rule to suites on the same head. Historical Cargo-topology recovery observation: the older graph published GHA Cargo / Cargo lane gate alongside CI gate; that is not the current workflow inventory. In that observed recovery, auto-merge remained blocked after the newest of three suites reported a successful CI gate; it fired only after the last remaining suite reported that context successful. Treat each suite that publishes a required context as independent recovery evidence, because neither the newest nor the latest-completed result was observed to supersede the others. In this recovery, gh pr checks reported pass while guarded merge remained blocked, so an attempted merge that stays blocked is stronger evidence than that flattened view. Inspect the suite records directly when the rollup and merge policy disagree:

​
Nightly Lanes

Everything pull-request CI deliberately does not run executes daily and on demand (.github/workflows/nightly.yml) on GitHub-hosted runners:
  • the full workspace unit lane (make test-unit) and integration-fast lane (make test-int), covering every crate rather than the changed packages
  • the deterministic e2e-fast lane
  • the 300-member/150-peer dense Mob topology stress (mob-dense-topology.yml)
  • bounded TLC machine verification (make machine-verify)
  • the Python + TypeScript SDK suites, unconditionally
  • clippy across all targets and all features for the whole workspace
  • feature-matrix lint and tests, minimal-feature and surface-modularity tests
  • e2e-system, the full Web SDK and WASM browser contract suites, release packaging dry-runs, the security audit, and the MobKit docs mirror lag
  • the whole BuildBuddy/Bazel graph (buildbuddy.yml in full-fresh mode): prebuild, fmt/static, the native nextest lanes, governance (seam-inventory, rmat-audit, machine authority), SDK suites, wasm-check, minimal-feature, feature matrix, and audit

​
Release Validation

The tag release workflow requires successful exact-main CI for the release commit and downloads that run’s schema-4 attestation (legacy schema 1-3 attestations are still accepted for older commits). It then runs the whole BuildBuddy/Bazel graph in full-fresh mode (release_validate_buildbuddy_full) as the release validation gate; the Cargo validation lane remains the fallback for manual dispatches on the GitHub-hosted backend. It still builds and publishes the platform-specific release artifacts from the attested source; Windows binaries are cross-compiled and packaged with cargo-xwin (clang-cl, lld-link, the Windows SDK) on a hosted Ubuntu runner, then executed for verification on a windows-latest runner. Registry publication remains GitHub-hosted. The repository pre-push dispatcher follows the same identity rule. A complete successful hook run writes a local Git-tree-keyed evidence stamp. A later push of the identical tree, including its release tag, validates the pushed object and checked-out HEAD identity and then reuses the complete gate result. BuildBuddy remains an opt-in developer backend, the nightly full graph, and the release-tag validation and Linux/macOS packaging backend. .github/workflows/buildbuddy.yml is workflow_call-only; nightly.yml and release.yml call it, ci.yml never does.

​
Doctor

Run:
The doctor checks the API key, pinned bb CLI, generated Bazel files, selector behavior, lane isolation, and docs wiring without printing secrets.

​
Multi-Agent Work

Separate Git worktrees are already isolated by path hash. If multiple agents share one checkout, set a distinct RUST_LANE_ID per agent when you want stable warm local output roots.

​
See Also