​

Multi-Host Mobs — Architecture & Implementation Plan (v4)

Status: adjudicated v4 (2026-07-07, at HEAD 0.7.22 / da467fca8). Every substrate claim below was re-verified against that HEAD (file:line anchors are from it). This document is the merged, authoritative plan. It supersedes the v1 sketch (2026-06-03), the v2.1 adversarial adjudication (2026-06-15), and the pasted “RMAT-aware extension plan” draft; the v3 “leaderless peer fabric” concept is a separate track, not this plan (see Adjudication history). Companion doctrine: docs/architecture/meerkat-dogma.md. Machine registry: canonical_machine_schemas() in meerkat-machine-schema/src/catalog/mod.rs.

​

1. Goal

• Agents communicate by identity (AgentIdentity), never knowing whether a peer is local or remote.
• Users observe and control every member — including remote and remotely-spawned members — through the same console/API surfaces: list, status, history, live events, send, cancel, retire, wire.
• Mob membership/topology stays MobMachine authority. Comms delivery stays meerkat-comms authority. Session execution/history stays runtime/session authority. MobKit stays a projection layer.

Host A: A            (controlling host: runs the MobActor / MobMachine)
Host B: B1, B2, later B21, B22   (B2 spawns B21/B22 onto Host B)
Host C: C1, C2, C3

Edges:  A<->B1  A<->C1  A<->C2  B1<->B2  C1<->C2  C2<->C3

​

2. Adjudication history (do not re-litigate silently)

​

3. Constraints (v1, corrected wording)

• Realms are not distributed. A member’s sessions/events/credentials persist in the member host’s local realm. Cross-host reads are bridge-served projections, never remote store handles.
• One process hosts many members; one TCP ingress port per host process (not per agent, not per member).
• No new authority owners. No parallel roster, no second topology counter, no host-local fence rotation, no comms-owned route authority. Every new semantic fact folds into an existing machine or a catalog-generated scoped authority.
• Protocols are separated; transport is shared. Peer data (MessageKind) and supervisor control (BridgeCommand over the supervisor.bridge intent) already share the comms envelope, listener, and inbox — that stays. Separation is at the protocol/authority level: typed payloads, distinct admission authorities, one ingress. (The draft’s “do not merge the planes” wording is retired; read literally it contradicts shipped reality — meerkat-runtime/src/comms_drain.rs:1687 dispatches bridge commands out of the ordinary drain.)
• The peer envelope does not change in v1 (see D5-adjacent rationale in §7.1).
• MobKit consumes Meerkat surfaces; it grows no roster, directory, or authority.

​

4. Decisions

• Plane (a), host↔mob: a bind ceremony mirroring BindMember (BridgeBindPayload, meerkat-contracts/src/wire/supervisor_bridge.rs:758-766): one-time bootstrap token → durable host authority record fenced by the supervisor authority epoch. Possession-proof + Ed25519 identity; all-or-nothing per host. Comms bootstrap tokens are never principal auth.
• Plane (b), principal→mob: typed ControlScope grants owned by MobMachine (grant lifecycle = machine facts) and enforced through a sealed resolved policy at the controlling host’s command dispatch — the ToolExecutionPolicy/ExecutionPolicyGatedDispatcher pattern (meerkat-core/src/tool_execution_policy.rs), including its fail-closed UnresolvedInherit discipline. Bridge commands are only ever sent by the already-authorized supervisor, so plane (b) enforcement lives entirely on the controlling host (MobCommand admission + operator surfaces), not on member hosts.

​

5. Authority model

​

6. Machine deltas (catalog first — Rule: no production-only semantics)

​

6.1 MobMachine — hosts and placement

mob_hosts:                 Set<HostId>
host_public_keys:          Map<HostId, PeerSigningKey>
host_endpoints:            Map<HostId, PeerAddress>
host_authority_epochs:     Map<HostId, u64>
host_bind_phase:           Map<HostId, HostBindPhase>       // Requested | Bound; absence = unbound
host_capabilities:         Map<HostId, HostCapabilityFlags> // durable_event_replay, autonomous_members, hard_cancel, protocol range
member_placement:          Map<AgentIdentity, HostId>       // ABSENT = controlling host (local). Present = remote.
member_materialization_failures: Map<AgentIdentity, String> // mirrors member_restore_failures

​

6.2 MobMachine — wiring install obligations (D4)

pending_route_installs: Set<RouteInstallObligation>   // { edge: WiringEdge, host: HostId }

​

6.3 MobHostBindingAuthority — new catalog-generated scoped authority (host side)

supervisor binding:     peer_id, epoch, bind phase        (mirrors member-side SupervisorBridgeCommandAdmission facts)
materialized members:   Map<AgentIdentity, { generation, fence_token, session_id }>
dedup memory:           last-materialize tuple per identity (idempotent replay; stale tuple → typed reject)

​

6.4 MeerkatMachine — member-side admission extension

​

6.5 MobMachine — subscriptions and grants

• Event subscription: the adjudication seam exists with two outcomes — AuthorizeAgentEventSubscription { agent_identity, session_id } / RejectAgentEventSubscription { reason: MemberNotFound | NoSessionBinding } (mob_machine.rs:1323-1324). Add the third outcome AuthorizeExternalAgentEventSubscription { agent_identity, host } guarded on member_placement/external_peer_edges. Extend AuthorizeAllAgentEventSubscription to carry the external member set next to session_bound_runtimes — today the mob-wide stream silently omits external members (AuthorizedMobEventRouter filters to session-bound, meerkat-mob/src/runtime/event_router.rs:48, handle.rs:2582-2653); this plan kills that silent cap.
• Grants: operator_grants: Map<PrincipalId, GrantRecord { scopes, expires_at_ms }> with GrantOperatorScopes / RevokeOperatorScopes inputs. Expiry is data checked at the enforcement seam (shell reads the clock; the sealed policy compares — mirrors how ToolExecutionPolicy keeps policy resolution out of the machine while lifecycle stays machine-owned).

​

7. Wire & protocol deltas (meerkat-contracts; one versioned protocol)

• ReadMemberHistory { supervisor, epoch, protocol_version, from_index, limit } → BridgeReply::MemberHistoryPage (mirrors SessionHistoryPage; transcript snapshot pages).
• PollMemberEvents { supervisor, epoch, protocol_version, cursor: MemberEventCursor | Tail, max, wait_ms } → BridgeReply::MemberEventsPage { generation, events: Vec<EventEnvelope>, next_seq, watermark }. Long-poll: wait_ms bounded well under the bridge transport budget. Served from EventStore.read_from when durable_event_replay, else from a bounded live buffer with typed StaleCursor on overrun.
• HardCancelMember — variant exists on the wire today but every receiver rejects it and nothing sends it (provisioner.rs:3297-3312; comms_drain catch-all). v1 implements the receiver arm through the machine-admitted hard-cancel path and a production sender, gated by the Cancel scope.

• BindHost { supervisor, epoch, protocol_version, expected_host_peer_id, expected_address, bootstrap_token } → capabilities reply (mirror of BindMember).
• MaterializeMember { spec, agent_identity, generation, fence_token, budget_seed, launch_mode } → MemberMaterialized { member_pubkey, member_peer_id, advertised_address, session_id }. Idempotent on (agent_identity, generation, fence_token): replay returns the recorded result (the BridgeDeliveryOutcome::Deduplicated precedent); a lower tuple is a typed StaleFence reject.
• ReleaseMember { agent_identity, generation, fence_token } — retire/teardown on the owning host; also the reconciliation verb for orphans.
• InstallPeerTrust { agent_identity, peer: TrustedPeerDescriptor } / RemovePeerTrust { ... } — realize RouteInstallRequested on the owning host through the member’s machine-gated apply_trust_mutation seam (never direct TrustStore writes).
• HostStatus {} → materialized-member inventory + health (feeds reconciliation and reachability projection).

​

7.1 Why the envelope stays untouched (recorded so it is not re-proposed)

​

7.2 Host bind ceremony (plane a)

1. Operator starts rkat mob host on Host B. The host loads/mints its host keypair (Keypair::load_or_generate in the host identity dir), binds the acceptor, and writes a host binding descriptor (the --comms-binding-out shape, host flavor: {kind: "host", address, ed25519 public_key, bootstrap_token}).
2. Operator (or automation) hands the descriptor to the controlling mob: BeginHostBind → routed RequestHostBind → bridge BindHost with the one-time bootstrap token, expected identity, expected address.
3. Host validates via MobHostBindingAuthority (token, sender, address — the validate_bind_request shape, comms_drain.rs:879-965), records the supervisor binding at the offered epoch, replies capabilities.
4. CommitHostBind records {HostId, pubkey, endpoint, capabilities, epoch} in MobMachine. Supervisor rotation fan-out (already fail-closed with pending-rotation memory) extends its recipient set to bound hosts.

​

7.3 Placement and materialization

​

7.4 Events, history, completion, cancel — closing the observation gap

• History: console mob/member_history (new RPC/REST/MCP surface, by AgentIdentity) → controlling host: local member ⇒ existing session read; remote ⇒ ReadMemberHistory proxy. Same page shape either way.
• Events: the controlling host runs a poll pump per remote member (schedule-driver-style loop) driving PollMemberEvents from the last cursor, wrapping results as AttributedEvent { source, source_fence_token, role, envelope } (the existing merge item, meerkat-mob/src/event.rs:509-518) and feeding AuthorizedMobEventRouter next to the local SelectAll. Dual cursor: the source (generation, seq) is preserved on each item; the merged stream’s ingest order is the router’s own. Console UX is unchanged (mob/stream_event push from the controlling host). Pump lifecycle follows the third subscription outcome (§6.5).
• Completion: falls out of events — terminal AgentEvents arrive through the pump; wait_one/wait_all/flow-step completion for remote members consume the merged stream (the spawn_turn_completed_reply degradation path is upgraded to it). DeliverMemberInput keeps replying at admission (unchanged; the reply deliberately carries no turn payload).
• Hard cancel: receiver arm lands (machine-admitted), capability flips, Cancel scope gates it at the controlling host.

​

7.5 Reachability (projection)

​

8. Scoped control (plane b)

enum ControlScope { List, ReadHistory, SubscribeEvents, SendCommand, Cancel, Retire, WireTopology, AdminHost }

• Grants are MobMachine facts (§6.5), keyed by PrincipalId (comms-principal = PeerId; local operator contexts injected — Gotcha #19: operator authority is injected, not ambient; MobToolAuthorityContext + seal at meerkat-core/src/service/mod.rs:491,511 is the vocabulary to extend).
• Enforcement: a sealed ResolvedControlPolicy checked at the controlling host’s two chokepoints — MobCommand admission and the operator surfaces (RPC/REST/MCP mob handlers + agent-facing mob tools). Deny = typed ScopeDenied { required, presented }. Fail-closed: an unresolvable principal has no scopes.
• Defaults: the owning session (owner_bridge_session_id) holds implicit full scope — single-user CLI behavior is unchanged. Every other principal is default-deny until granted. Scope semantics honored end-to-end: SendCommand without ReadHistory can drive but not read; SubscribeEvents without Cancel can watch but not stop; WireTopology is distinct from SendCommand; AdminHost (bind/revoke hosts) is distinct from everything.
• Ordering rule: no remote read/mutation surface ships before this lands (phase 5 precedes phase 6).

​

9. Failure semantics (typed, enumerated)

​

10. Phases (exit gates, no calendar)

1. Adjudication + catalog deltas. This document is the phase-1 artifact. Land §6 in one catalog change-set (fields, inputs, effects+dispositions, invariants, MobHostBindingAuthority DSL, MeerkatMachine admission coverage) + §7 contracts (V4 commands, replies, causes, cursor, projections). TLC bounds: 2 hosts × 3 members.
2. Host bind + acceptor + host role. D1 demux in meerkat-comms (registry install/remove from machine effects only; mandatory peer auth; acks signed per-member; pairing branch host-scoped); §7.2 ceremony end-to-end; rkat mob host daemon (D6); host binding descriptor.
3. Placement + materialization. §7.3 end-to-end including B2→B21/B22 on Host B; HostMaterialized binding variant; ExternalBackend stub deleted; idempotent retry; orphan reconciliation; revival integration.
4. Cross-host wiring install. §6.2 obligations; placement-aware transport selection; wire_members_batch local-only rejection removed; unwire symmetry; fail-closed partial install with durable retry.
5. Control scopes. §8 grants + sealed policy at both chokepoints; ScopeDenied; owner-implicit-full default. Must land before phase 6 ships any remote read.
6. Events / history / completion / cancel. §7.4: DSL third outcome + all-subscription external fan-out; ReadMemberHistory + PollMemberEvents + pumps + router merge with dual cursors; completion consumption; HardCancelMember receiver + sender; bridge single-flight relaxed; BRIDGE_CLASSIFIER_FILES extended.
7. Surfaces. RPC/REST/MCP/SDKs: by-identity member_history, member event streams for remote members (existing mob/stream_event now covers them), host register/status, route-install status, grant management (rkat mob grant/revoke, host verbs). RPC stays the console↔controlling-host transport; rkat-rpc --tcp is never a host↔host control path (one control plane: the bridge).
8. MobKit projection. Remote/local members project into the existing console identity records; remote history/events consumed via the new Meerkat surfaces; console contracts stable; no MobKit-owned roster/directory (verified by review, not assumed).

​

11. Test matrix (lane-mapped; deterministic lanes are the CI ratchet)

• mixed A/B/C topology: identity-routed send local↔remote both directions; B1↔B2 same-host inproc selection
• many identities, one acceptor: demux by to, per-member ack signing, misaddressed reject, unregistered-identity reject
• mandatory peer auth on acceptor cannot be disabled; envelope byte-compat pin (no signed-region drift)
• host bind ceremony: token single-use, sender mismatch, address mismatch, capability capture, epoch record
• remote spawn ladder: B2→B21 full walkthrough; materialize retry deduplicates; stale-fence materialize rejected; orphan reconciliation releases at stale fence; revival classification on materialize failure; Broken refuses retry
• wiring: partial install leaves obligation + unusable edge fails closed; retry drains; unwire removes trust and subsequent delivery is rejected; batch wiring with mixed placements
• events: durable cursor resume across pump restart AND member respawn (generation bump resets seq domain); mob-wide stream includes remote members; completion for remote members via merged stream; long-poll does not block lifecycle commands; StaleCursor on ephemeral overrun
• history: remote page read == local page shape; scope matrix — each ControlScope grants exactly its verbs, ScopeDenied carries required/presented; work-only cannot read history; read-only cannot cancel
• rotation: bound hosts in fan-out; pending-rotation retry with a host that rebound to current authority
• TLC (make machine-verify) at 2×3 bounds green for all new state

​

12. Non-goals (v1) and v2 seeds

​

13. Implementer gotchas (verified traps)

1. BRIDGE_CLASSIFIER_FILES is a hardcoded allowlist — extend it for every new bridge-reply consumer (§7.4).
2. RuntimeHostInfo must stay anti-authority (pinning test at wire/host.rs:181-210) — do not enrich it with host/placement authority facts.
3. DSL wiring inputs are plural; the singular names are bridge commands. Do not invent WireMember DSL inputs.
4. No member-side fence/epoch validation exists on delivery today — the host-side StaleFence checks in §6.3 are new work.
5. ExternalBackend (provisioner.rs:2580) is an empty stub — delete, don’t extend.
6. MobSupervisorBridge single-flight request_lock will serialize long-polls against lifecycle commands unless relaxed (§7.4).
7. Trust rows are never seeded from config and persisted seeds are rejected at startup (comms_runtime.rs:1620-1632) — all trust flows through apply_trust_mutation under machine authority; the acceptor registry follows the same rule.
8. Three sequence domains exist (session-task seq, per-RPC-stream seq, durable store seq) — only the durable store seq is a cursor; never leak the others into the wire cursor.
9. Re-fencing on re-acquire is intentional (fencing mechanism, not churn) — do not “fix” monotonic fence bumps on host rebind.
10. Effects need typed dispositions (seam-inventory) and commands need typed classification (runtime_alphabet_parity) — budget for these in every machine delta, they are CI failures, not review notes.