Skip to main content
Meerkat uses executable machine definitions for state that must have one semantic owner. The goal is simple: for any important lifecycle question, there should be one authoritative answer to what state exists, which transitions are legal, and what effects follow.

Coordination Boundaries

  • WorkGraph is the durable shared-work ledger. It owns dependency readiness, work claims and leases, evidence, and terminal work verdicts.
  • Schedule is the time authority. It owns trigger calculation, recurrence, occurrence claims, delivery attempts, and occurrence outcomes. A scheduled wake does not decide whether WorkGraph work remains ready or complete.
  • Detached Jobs own durable background execution, while Runtime Delivery owns delivery identity, sequence, replay, and application progress. A job terminal receipt does not by itself prove that the target runtime applied the result.
  • Peer delivery accelerates discovery of durable work and wakes disposable execution shells. A peer send or handoff receipt is not a WorkGraph claim, Schedule occurrence verdict, or proof that shared work completed.
  • Durable claims are authorized only by their owning machine and store contract. WorkGraph authorizes work claims, Schedule authorizes occurrence claims, and peer FIFO removal requires the exact runtime admission receipt bound to its claim. Queue snapshots, counters, ages, correlations, and projections are observability facts and cannot authorize mutation.

Canonical Machines

The canonical registry is canonical_machine_schemas() in crates/meerkat-machine-schema/src/catalog/mod.rs. MeerkatMachine and MobMachine are the two runtime kernels. The other canonical machines protect narrower domain facts: auth, approval and durable decisions, schedules and occurrences, detached jobs and runtime delivery, session document and turn admission, WorkGraph lifecycle, attention, and execution, scoped tool grants, and the writer fence of a host store realm. WorkGraph is an optional subsystem authority for durable agent commitments, dependency-aware claim state, and goal attention bindings. WorkExecutionLifecycleMachine owns the cross-system obligation for one execution attempt without absorbing either WorkGraph item truth or Mob Flow run truth. Its effects are realized mechanically by the Mob composition facade, and every launch, evidence, or closure outcome returns through a typed machine input before the binding advances. Two scoped authorities sit outside the canonical registry. SessionPersistenceVersionAuthority owns admission of released session envelope and metadata schema versions. Its catalog DSL lives at crates/meerkat-machine-schema/src/catalog/dsl/session_persistence_version_authority.rs, and its production expansion lives at crates/meerkat-core/src/generated/session_persistence_version_authority.rs. It is a small restore-time authority, not a persistence-currentness or recovery authority, and it has no entry in canonical_machine_schemas(). MobHostBindingAuthority (catalog DSL source crates/meerkat-machine-schema/src/catalog/dsl/mob_host_binding_authority.rs, production expansion crates/meerkat-mob/src/machines/mob_host_binding_authority.rs). It is the member host’s process-scoped, mob-keyed admission and dedup authority for host-addressed mob commands: supervisor bind/rebind/revoke, host-command admission, materialize admission/preflight with success-only dedup memory, release admission with recorded-disposal replay, and the remote turn-outcome journal. It follows the session_persistence_version_authority precedent — one shared DSL body expanded into both crates, pinned by a dedicated production-schema parity test (crates/meerkat-mob/tests/mob_host_binding_authority.rs) and by kernel tests instead of TLC — so it carries seam-inventory dispositions like any machine but has no poster, no generated kernel, and no entry in canonical_machine_schemas(). Principal control-scope grants (multi-host mobs §8) follow the ToolExecutionPolicy split. The MobMachine owns the grant lifecycle — the operator_grant_scopes / operator_grant_expiries facts, the GrantOperatorScopes / RevokeOperatorScopes transitions, and the in-machine revalidation of caller-proposed revoke partitions — while scope resolution lives shell-side in the sealed ResolvedControlPolicy (crates/meerkat-mob/src/control_policy.rs): private shape, one resolve() mint path, no serde, fail-closed to the empty scope set. Expiry is data: the machine never reads a clock; the enforcement chokepoint reads the wall clock once per decision and passes now_ms in as a parameter, so a restored mob’s expired grants stay expired with zero persisted derived state. Grant durability is a runtime-metadata record (MobOperatorGrantRecord) written only under GrantRecorded/GrantRevoked transition witnesses and replayed on resume through the machine’s own GrantOperatorScopes input — never a second enforcement source. Identity entries (a host’s own durable identities, as opposed to worker members; see Mob Architecture) are MobMachine facts too. The machine owns which members are entries (identity_entry_origins) and which open spawn ladders act for one (spawn_exec_entry_custody): a BeginSpawnExec that carries entry custody records the entry in the membership commit or the dormant seat that closes its ladder, in the same step that seats the member. The legacy adoption (AdoptLegacyIdentityEntry) decides from the shell’s raw observation of the member’s spawn row, and the shell journals an adoption before it commits. The replay of either row records the entry again (RecoverIdentityEntry, before the boot plan is sealed). The admission probe answers an entry’s name without custody IdentityMemberRejected, the worker reconcile helpers range over workers only, and the agent lane’s member admission (ResolveAgentLaneMemberAdmission) is a pure verdict the actor’s command admission realizes. The handle’s entry reads project the published machine state. A respawn of an entry’s member is a same-session rotation (below): the identity stays seated and an entry from its retirement’s start to its next incarnation’s seat, each one machine step, so a respawned entry no longer passes through a window in which it is neither seated nor an entry. The machine owns the respawn’s route: its fresh-life respawn arm (RespawnRunning, whose input Respawn names the identity) admits no entry (not_an_identity_entry), and its refusal twin (RespawnIdentityEntryRefused, in Running, Stopped and Completed) answers RespawnRefused with the reason IdentityEntryKeepsSession, so no fresh-life step runs for an entry in any phase an entry lives in. The fresh-life respawn’s committed archive (the topology-preserving ObserveMemberRetirementArchived arms) refuses an entry too, whatever the shell previewed. The actor previews the verdict and routes on its reason: an entry’s respawn is that rotation, a successor spec for one is answered typed, and the retire-then-spawn path previews it again before it retires anything (its successor never takes entry custody). One shell step still decides from the machine’s entry facts until the machine owns it: a seat with entry custody of a name a worker holds is refused MemberAlreadyExists over the published facts. No other shell code records, removes or derives an entry. The ci profile has no identities, so the hand-written audit specs/machines/mob_machine/identity_entry_audit.sh drives the entry arms in its runs: one records, refuses and replays entries, one takes an entry through the dormant seat, the live retirement, the rotation, the fresh-life respawn’s refusal (in Running and Stopped) and the destroy, and the others drive the declared apply below. The declared apply (MobHandle::apply_declared_roster; see Mob Architecture) is MobMachine’s too. DeclareIdentityEntry answers one verdict per declared identity (IdentityEntryVerdict), one arm per verdict, each guarded by the pure helper identity_entry_verdict over the declaration and the member’s facts, and each guarded to act only on the entries of the custody it declares, by an explicit check: a roster’s declaration (created_custody false) on an identity that is no entry or a Declared one, a store-owned update’s (created_custody true, since 0.9.9) on its Created entry, seated, owed its seat, hosted on its first use or awaiting its held creation, and, answered RecordOnly (its row alone changes, and its unarchive or settle seats the new spec), archived or awaiting its host’s preparation; an entry of the other origin is taken by no arm. A store-owned update is a declaration under created custody, so both custodies answer the same verdict for the same facts, and MobMachine routes it from its own step’s state. The declaration carries the shell’s raw observations: whether the host’s rule requires the identity’s preparation before its first seat (preparation_required, answered PreparationPending for a roster-owned identity never seated), the identity store’s intent rows, the session service’s durability, the role the session an unseated identity’s seat would resume was last built under (resume_session_role, so a seat under another role takes the migration declared from that role or is held), and the kind of the host’s first-boot request (first_boot), whose preparation the declaration’s own step ends when it carries the request out. Its facts are the unseated entries with their typed cause (identity_entry_unseated: SeatOwed, HostedOnFirstUse, HeldCreation, PreparationPending, Archived), the role hold (member_host_preparation_causes, cause RoleMigrationUndeclared, which refuses every wake as a first-boot preparation does, holds a Live local member’s run starts through MeerkatMachine’s HostPreparation hold reason and refuses the explicit resume of the member (answered held while the mob is still Stopped, before the resume prepares any member, and again at its rebuild), and which ends at a verdict that restores the role or in the step that commits the declared one, never at the verdict that declares the move, and with its entry wherever the entry ends, role_hold_belongs_to_an_entry) and the archived session of an entry’s retired member (identity_entry_archived_sessions). None of them has a journal row of its own: the replay derives an entry’s unseated standing and its archived session from its member’s retirement rows (the terminal row records whether the retirement archived the session, session_archived, and the replay passes whether the incarnation was hosted elsewhere, hosted_incarnation), and the next apply declares the rest again. Some row fields are journal-only facts, for the member lifecycle journal read (MobHandle::member_lifecycle_events), and the replay reads none of them: a retirement’s session, the entry it was the member of, its cause (the seal’s mark) and the rotation whose terminal step retired it, and a seat’s session and the rotation that seated it (a rotation’s terminal step, or rotate_member’s fresh life of an identity no member holds, which follows the generation the machine retains for the identity). The shell writes each from the machine’s state before the step or from the step’s journal effect; no machine input or effect field carries them. The seal of a complete roster (SealDeclaredRoster) marks the seated Declared entries it does not list (identity_entry_retiring) and removes the unseated ones, and leaves an entry whose spawn ladder is open to a later seal (marked_entry_has_no_spawn_in_flight), so no mark lands between a spawn rollback’s start row and its Retire step; a declaration that lists a marked entry again before its retirement started drops the mark. The shell retires each marked member, its start row records the cause, and its replay passes every such row (RecoverIdentityEntryRetiring, which marks only the identity’s seated Declared incarnation and takes a no-op arm for any other row). The end of an entry’s retirement records the session it archived, and ends the entry when the seal, a one-entry unlist or a removal marked it (a removal’s end also tombstones the name); an archive’s mark leaves the entry unseated, Archived; any other end keeps the entry unseated, owed its seat (hosted on its first use when placed or externally bound). The dormant seat passes its raw observation that the session is archived and revivable, and its guard binds an archived session only when it is the entry’s record; it takes a fresh session only once the record is let go (ForgetArchivedSession: the session was observed absent, or the host asked for a fresh life), and it never seats an entry hosted on its first use or awaiting its held creation. A seat ends a first-boot preparation only: a role hold survives it. A placed spawn holds an entry’s custody only for an entry hosted on its first use (placed_entry_custody_is_hosted_on_first_use). Applies are serialized by the apply-open fact (declared_apply_open): OpenDeclaredApply answers Opened, or Busy while an apply is open, and CloseDeclaredApply closes it in every phase. The shell closes it through the lease the opening step made, on the internal lane: no grant and no Shutdown step refuses the close. A mob reset ends every identity entry, and it commits only on an empty roster (ResetToRunning’s roster_empty guard): its preflight, AdmitReset, answers the reset’s other guards, changing nothing, before the retirement batch that empties the roster. It also ends every identity’s retained incarnation history, as a restart after the reset does (the resume replays only the events after the last reset): it folds each retained generation into the identity’s high-water (identity_generation_highwater), so the next incarnation still mints above every generation the identity used, and drops the retained generation, fence, profile and runtime mode together. The generation high-water audit’s reset run checks both, with a witness of the fold and one mutant per half. The audit’s apply runs (apply, legacy, resume, roles, hold, held and placed) drive the declare arms, the seal (which leaves a member whose spawn is in flight unmarked), the legacy adoption before an unlisted retirement, the dormant seat with and without an archived record, the retirement to its terminals, the reset over an emptied roster, the role hold against the wakes of every cause and the explicit resume, held creation, and a placed identity’s custody, remote commit, retirement and replay, with witnesses and mutants. Created identity entries (IdentityEntryOrigin::Created, since 0.9.9) are entries of the same map, recovered from the identity store at a start. The origin is the entry’s provenance and the custody of its desired record: it decides the recovery, which writer persists a change of that record, the name guards and the reservations below, never the member’s profile, labels, tools, grants or wiring. Since 0.9.9 a created entry can be placed, externally bound, held for its host’s creation or prepared by its host first, as a declared one can: created custody takes a Created entry owed its seat, hosted on its first use or held for its creation, and one awaiting its preparation only for the spawn of a construction held for it, on a local or a placed ladder (placed_entry_custody_is_hosted_on_first_use holds a placed ladder of either custody to an entry hosted on its first use). After the journal replay and before the boot plan is sealed, the shell sends one RecoverCreatedIdentityEntry per created entry row, with the row’s presence (RecoveredPresence: Present, PresentUnseated, PendingConstruction, Tombstone, Archived, Unreadable) and the session the identity’s member was last bound to in the replayed journal, and MobMachine decides from them and its replayed facts, in Running or Stopped. A present row records a seated local member a Created entry, or, when the replay left the identity without a member, records the entry unseated (SeatOwed) with that session as its revival record; a pending row records it unseated for its host’s preparation (PreparationPending), and a pending row of a held construction whose member the replay seats (its publication committed before a crash) records the member a Created entry; an archived row records it unseated (Archived); a removed or archived row whose member’s retirement the replay holds in flight records the removal or the archive in flight (identity_entry_removing, identity_entry_archiving), one mark per entry, so a second row contradicting a recovered mark reserves the name as unreadable; a removed row records the name’s tombstone (identity_entry_tombstones); and an unreadable row, or one the replay contradicts (a tombstone of the name, or a spawn ladder open for it, included), reserves the name (identity_entry_unreadable). No arm records an entry for a tombstoned name, and no replay seats a member under one, or under an archived entry. The store holds an archive’s row beside its member’s roster row while the archive’s retirement runs: the replay, which comes first, seats that member and the scan then marks the archive in flight; a replay that came after the scan recorded the entry archived is refused, so MobMachine does not depend on that order. Nor does a Declared row’s replay replace a Created entry (a store-owned identity never turns roster-owned): replayed first, a Declared entry stands and the scan’s row of it reserves the name; scanned first, the replay of a Declared row over the Created entry is refused. A Created row may replay over a Declared origin: the journal records no end of an entry that ends unseated (the seal’s or a one-entry unlist’s), so a later store-owned incarnation’s row takes over the stale origin of a roster-owned identity whose entry ended. MobMachine does rely on the boot’s order for a row pending its host’s preparation: the start replays the journal (seed_mob_authority_sync_from_events, then the placed carriers’ recover_current_committed_placed_carrier_signals) before it scans the created entry rows (recover_created_identity_entries). Scanned first, a pending row would record an identity without a member awaiting its preparation, and a replay that seated the identity after it would drop that cause: the start would neither settle the row of a published construction nor reserve the name a member contradicts. The replay reserves the name of a member row whose origin token this build does not know the same way. A removed entry’s name is never an entry or a member again; an unreadable name keeps what the replay recorded for it (a Declared entry, a member) and takes nothing new: the admission probe answers IdentityNameReserved (no spawn ladder opens, and every ladder close, the membership commits, the remote commit and the dormant seat, refuses a reserved name: a ladder opened before the reservation seats no member, and the shell aborts it with the name’s admission answer, MobError::IdentityMember), the worker reconcile helpers skip it, the worker ensure refuses it (name_not_reserved on EnsureMemberRunningMissing, which the shell answers with the name’s admission answer), the declared apply refuses it typed (RefusedNameTombstoned, RefusedNameUnreadable), the legacy adoption answers NotAdoptable, and the agent lane’s admission answers the name’s member IdentityMember. Custody of one origin never takes an entry of the other (IdentityMemberOfAnotherOrigin): entry custody takes a Declared entry’s name only, and created custody (BeginSpawnExec’s and ProbeMemberAdmission’s created_custody, recorded in spawn_exec_created_custody) a Created entry only, never both; the dormant seat that closes a created ladder takes the entry’s revival record and keeps the member a Created entry, and seats only an entry owed its seat. A roster’s declaration never declares over a Created entry (RefusedNameOwnedByCreated, which the shell answers as a typed note, a quiet steady state), and the seal unlists readable Declared entries only. A reset keeps the reserved names, which the identity store holds, and drops created custody; the destroy clears both. The invariants tombstone_is_never_an_entry, entry_custodies_are_disjoint and archived_entry_is_store_owned hold these facts (the earlier invariant unseated_created_entry_is_preparing_or_owed is retired: a store-owned update’s declaration records a created entry hosted on its first use or held, and the created identity audit’s scan run checks what the scan alone records). The shell re-seats an entry owed its seat once the mob runs, as the handle’s own command through that created custody, from the row’s spec (or, for a row written before rows carried their spec, from its member’s last spawn row with its roster labels), and MobMachine’s guards decide each seat. Store-owned identity changes are MobMachine’s too (since 0.9.9; see Mob Architecture). Each is decided in its commit step, from the shell’s raw observations, before the shell writes the change’s store transaction, and a refusal leaves the change’s prepared authority unapplied: AdmitIdentityChange answers the change’s admission (IdentityChangeAdmissionKind) from the store’s receipt of the operation, the consent’s form, whether a decision’s permit binds the change and admits a commit now, whether the requester is the host operator, whether an agent requester’s grant is current at this final sink, whether the spec the change was decided against is the identity’s (StaleSpec otherwise) and the host’s changes hold (identity_changes_held, set by SetIdentityChangesHeld); RecordCreatedIdentityEntry records a new store-owned entry, unseated with the cause its helper derives from the request’s facts (whether its host’s rule requires its preparation, its placement, a held creation) in identity_entry_verdict’s precedence, under a free name only (CreatedIdentityRecordVerdict; an unlisted roster-owned identity’s archived session record holds its name); and RemoveIdentityEntry, ArchiveIdentityEntry and UnarchiveIdentityEntry answer one identity’s removal, archive and unarchive (IdentityEntryChangeVerdict). A removal or an archive of a seated entry marks its ending (identity_entry_removing, identity_entry_archiving) and the end of its member’s retirement consumes the mark, tombstoning the name or leaving the entry unseated, Archived, on its archived session; the destroy’s retirement ends the mark with the entry. An unseated entry is tombstoned at once, a held construction of it ending in the same step (an accepted activation is in flight until its publication ends), and archived at once unless it awaits its host’s preparation, whose archive is refused (PreparationPending, the change’s typed refusal host_preparation_pending). At most one ending mark lands on an entry, only on a seated one (entry_ending_marks_are_seated_and_disjoint). UnlistIdentityEntry unlists one Declared entry, decided on that identity’s own facts (UnlistEntryVerdict): its seated member’s retirement marked as the seal marks one, an entry no member holds ended at once, one whose member step is in flight deferred, and nothing else read. Each answers one arm per verdict through a pure helper, with a MobNotRunning arm per other phase. MobMachine keeps the mob’s owned edges too (since 0.9.11): one custody map, identity_edge_owners, from each edge an identity change declared to its owner, the identity whose change declared it. DeclareOwnedIdentityEdge declares an edge a store-owned identity’s change lists, to an identity of either custody (OwnedEdgeDeclareVerdict: a second declaration of an owned edge gives no custody, and a peer that is no identity is PeerUnknown); EndOwnedIdentityEdge ends one by its owner’s change or by the host’s policy reload that denies it (OwnedEdgeEndCause), naming its owner (OwnedEdgeEndVerdict: another identity’s end is NotOwner); a removal ends its identity’s edges, whichever end owns them, in its own step (RemoveIdentityEntry), and the reset and the destroy end them all; AdmitMemberUnwire refuses a plain unwire of an owned edge in force (MemberUnwireAdmission); and the boot recovers each edge from its owner’s row (RecoverOwnedIdentityEdge). The map is the edge’s desired record. An edge is in force while both its ends are identity entries (the helper owned_edge_in_force, and its complement owned_edge_out_of_force): only then does the shell realize it (wiring_edges) once both its ends are seated, each realization (a seat’s, a spawn’s or a pass’s) reading MobMachine’s verdict before its wire (AdmitOwnedEdgeRealization, OwnedEdgeRealization), and only then is it refused a plain unwire. An archive of an end leaves the edge owned and unrealized; an unlisting of an end leaves it owned and out of force, so a worker that takes the end’s free name is no end of it, and the end’s re-listing brings it back in force. The invariants owned_edge_owner_is_a_store_owned_end and owned_edge_joins_no_tombstone hold these facts. An import restores the edges of the rows it restores as the boot recovers them, after the boot plan’s seal too (RestoreOwnedIdentityEdge, OwnedEdgeRestoreVerdict), and writes each restored row with only the edges it restored (an edge another identity owns meanwhile is not the row’s): the custody map an import leaves is the one the next start reads. MobMachine decides an adoption into store custody too (since 0.9.11): AdoptIdentityIntoStore takes a roster-owned identity (Roster: a Declared entry seated on a local member bound to a session, or owed its seat) or a worker member (Worker: seated, with no entry, local and bound to a session, spawned with no profile or model override (spawn_overridden), at the generation its host observed) into store custody as a Created entry, its member’s runtime, generation and session untouched. A pure helper, identity_adoption_verdict, decides over the name’s facts, one arm per verdict (IdentityAdoptionVerdict): a member step in flight is AdmissionUnsettled, an open declared apply DeclaredApplyOpen, a stale generation PredecessorMismatch, and any other shape no adoption takes NotAdoptable, with its reason (IdentityNotAdoptableReason, a worker’s override among them: ProfileOverride); the shell then declares the member’s current edges to identities owned edges of the adopted identity in the same step. MobMachine decides the downgrade cycle too (since 0.9.11). PurgeIdentityRows admits the host operator’s purge of the mob’s rows of schema version 2 and later only under held identity changes, and only when the mob’s scan read any such row (rows_present; none is NothingToPurge); the step’s store transaction removes the rows when their digest is the export’s, and the machine records only identity_rows_purged, which refuses every later identity change (IdentityChangeAdmissionKind::RowsPurged) until the mob restarts. A start that reads no row of a Created entry’s name (RecoverRowlessCreatedEntry) reserves it, unreadable and rowless (identity_entry_rowless), so nothing acts on it and no seat binds it; the invariant created_origin_without_row_is_unreadable holds this. ImportIdentityRecord decides each exported record, one arm per verdict (IdentityImportVerdict): the record comes back for a member seated under its name in place (a plain member only on the session the record names), for an unseated name owed its seat on its archived session, as recorded for a pending, an archived or a removed name, or by the adoption’s step for a roster’s adopted identity whose roster declared it again; a name another holder keeps is reported and left, and a rowless entry’s name a member holds now is released to that member. The host’s preparation before an identity’s first seat is MobMachine’s for either custody. BeginPreparationConstruction holds an attempt of a constructed preparation (identity_entry_constructions_held); SettleHostPreparation settles a preparation without an attempt (the entry is owed its seat; a constructed one is refused ConstructionRequired) or accepts the activation of exactly the held attempt it names, decided before the attempt’s deadline and binding its construction, moving it to identity_entry_constructions_activating; and EndPreparationConstruction ends an attempt by its abort, its deadline, a refused construction or a failed publication, an activating one only by a failed publication. A spawn ladder takes an entry awaiting its preparation only while a construction is open for it, held or activating (the construction’s own spawn opens its ladder at its publication, after the settle accepted its activation), and the membership commit that publishes it requires its accepted activation and ends the attempt. At most one attempt of an identity is open (construction_is_held_or_activating). The ci profile has no identities, so the hand-written audit specs/machines/mob_machine/created_identity_audit.sh drives the created entries’ arms in fourteen runs, in two parts. Its recovery part has four:
  • The scan run scans a row of every presence, with and without a revival record, in Running and Stopped, until the boot plan’s seal. The rows are for a replayed seated member (with or without its Declared entry’s row, replayed before or after the scan; the member’s own replay never follows the scan of its row pending its host’s preparation, the boot’s order, which the names run keeps too) and for an identity without a member. The run reaches every outcome, the contradicted rows included, exactly one arm answers every row, and no step turns a Created entry Declared.
  • The reseat run opens, aborts and seats spawn ladders with every pair of custodies over the scanned rows, alongside the worker ensure, the worker reconcile and the destroy. It checks that no ladder opens for a reserved name, that a ladder’s custody matches its name’s origin, that a seat under created custody binds the entry’s revival record, that a row of a name an open ladder acts for reserves it, and that no worker ensure records a reserved name for a spawn.
  • The names run drives the declared apply, its seal, the legacy adoption, the agent lane’s admission and the admission probe over Created, removed and unreadable names. Exactly one arm answers each, and the seal never touches a Created entry.
  • The placed run scans the row of a member seated on a member host.
Its changes part (0.9.9) has ten:
  • The admit run drives every change’s admission from every fact, over the only facts it reads (the composition authority, the changes hold, a running and a stopped mob): exactly one arm answers each. The admission changes nothing, so no other run fires it.
  • The commit run drives the record of a name from every request’s facts, the changes hold and a seat or membership commit under created custody: exactly one arm answers each record, a new entry takes only a free name (no entry of either custody, member, ladder, reservation or archived session record holds it), its cause follows the declared apply’s precedence over its facts, and an entry held for its creation is never seated Dormant.
  • The update run declares the same change of a roster-owned and a store-owned identity, owed their seats or seated under the same spec: exactly one arm answers each, and both answer the same verdict for the same facts.
  • The removal run removes a roster-owned identity through its retirement (or ends it by the destroy with its mark) and a store-owned one unseated, seated through its retirement, or archived: every removal’s end but the destroy’s tombstones its name.
  • The archive run archives a seated store-owned identity through its retirement and unarchives it onto its archived session: an archived identity keeps its name and is never seated, an identity awaiting its preparation is never archived, the update of either changes its row only (RecordOnly), and a roster-owned identity’s archive is refused.
  • The preparation run drives the start, settle and end of held constructions of either custody with their ladders (the construction’s own ladder opening after its activation was accepted included), seats, membership commits, archives and removals, firing the attempts’ arms that change state: an entry awaiting its preparation is seated only by its accepted activation, no end of an accepted activation but its failed publication is enabled, a constructed preparation is never settled without an attempt, and its update changes its row only.
  • The prepverdicts run drives every arm of an attempt’s start, settle and end from every fact, over the facts their verdicts read: exactly one arm answers each.
  • The placement run seats a store-owned identity hosted on its first use by a placed or peer-only spawn holding its created custody.
  • The unlist run unlists a roster-owned identity through its retirement beside a store-owned one: exactly one arm answers each unlist, which decides on its identity alone, and no record takes the name of an entry or of the unlisted identity’s archived session record.
  • The recovered run replays a seated store-owned identity, its Created entry row included, and scans its one row, a removal’s or an archive’s (the store keeps one row per identity), before or after the journal replay. It carries the recovered mark’s retirement to its terminal, running or stopped: no replay seats a member under an archived entry, and the boot plan’s seal, a stop and a resume leave an archived identity archived.
The audit seeds defects of the invariants above, the action properties and the guards a reachable state needs, and refuses each with the violation it names (not every invariant has a mutant of its own); three mutants (created custody kept local, a recovered removal left unmarked, an accepted activation’s own ladder refused) are refused by making a witness unreachable. Member rotations are MobMachine facts. A pending rotation records its identity’s next session (member_rotation_session_ids), the endpoint it seats (member_rotation_peer_endpoints) and its target (member_rotation_targets): FreshSession (a reset, a deletion) seats the next incarnation on a new session, SameSession (a respawn, a declared role migration) on the member’s own. A same-session rotation starts the incarnation’s retirement in its own step (BeginMemberRotation, target SameSession) and keeps the binding; its retirement releases the runtime registration and keeps the session (disposal KeptForSuccessor, which no plain archive takes), and its terminal (the SameSession arms of CommitMemberRotation) retires the old incarnation and seats the next one on that session in one step, so the identity is never unseated. A releasing retirement of a member whose same-session rotation stands is refused: its re-drive preserves the binding. A same-session start takes the shell’s observation of the kept session (session_archived), which its session_not_archived guard refuses. The retirement’s start row journals the rotation, and its replay records the rotation again (RecoverMemberRotationIntent), so a mob restarted mid-rotation finishes it with the rotation’s own terminal and no host record, while the plain archive waits; an admitted destroy beats it. The start does not re-drive the FreshSession rotation of a member its host marked host-pending: the host’s re-issue finishes it. The start, the actor’s own re-drive and the re-drive of a caller other than the host (redrive_retirement_unless_held_for_host) decide that hold by one shell predicate over the machine’s facts, in the actor’s own step: host-pending and a FreshSession target. A same-session terminal keeps a host’s first-boot preparation (member_host_preparation_causes, cause FirstBootRow): only the host’s seat step ends it, so the successor of a host-pending member is seated Dormant and every wake of it is refused until then. Under the member’s own role the same-session terminal keeps the member’s endpoint and its wiring and external peer edges, which the replay of its MemberRetired row ends, so its journal batch records them again after the next incarnation’s row. Every same-session start records the role it starts with (profile_name, held in member_rotation_profile_names while the rotation stands, recorded again by the replay and dropped with the rest of the intent), and both same-session terminals refuse a grant of any other role (grant_takes_the_started_role) and take the role change from it: the start’s retirement prepared the member’s topology for that role. A same-session successor under another role runs under that role’s comms name: the start records an endpoint only for another role, under the member’s own peer id (endpoint_only_for_another_role, endpoint_keeps_the_members_peer_id, and the same guards on the replay), the shell’s retirement unwires the member’s peers’ trust as a plain retirement does, and the terminal ends the member’s edges and installs the recorded endpoint (or, with none recorded, owes one until the successor’s first wake registers it). The invariants same_session_rotation_keeps_its_binding and rotation_started_role_is_a_same_session_rotations hold the started role exactly while a same-session rotation is pending. The release verdict (ClassifyMemberRelease) takes the incarnation a scoped release names and answers Stale for any other. The hand-written specs/machines/mob_machine/member_residency_audit.sh drives these arms in its rotate-same and recover-rotation runs, and replays a committed same-session terminal’s rows, its kept edges among them, in its replay-same run. The retire of an identity that holds no roster incarnation (never a member, or already retired) is a MobMachine answer too, in every phase. The handle reads the roster only to choose the command: finding no incarnation, it asks the actor for RetireAbsent, the input the actor’s own retirement submits when it meets an absent member, and answers what the machine answers. RetireAbsent has an arm in each phase (RetireAbsentRunning, RetireAbsentStopped, RetireAbsentCompleted, RetireAbsentDestroyed, the last for a destroyed mob whose actor still serves its destroy’s retry). Each changes nothing while the identity maps to no runtime (identity_absent) and refuses an identity the machine still maps, such as a successor whose spawn committed after the caller read the roster. The handle answers that refusal as the spawn still in progress (MobError::LifecycleOperationPending, retryable). The input names no incarnation, so it never retires or cancels a later incarnation of the identity. The hand-written specs/machines/mob_machine/member_lifecycle_audit.sh fires the four arms and checks, in its scope run, that a RetireAbsent arm is enabled exactly when the identity maps to no runtime.

Consumed Checkpoint Completion Recovery

A live steer checkpoint can durably consume an input before its completion receipt is finalized. Its CompletedWithoutResult candidate is not a new RunResult, and a pending completion does not make the input abandoned. MeerkatMachine owns checkpoint correlation through the consumed input’s existing run association, independently of the latest run-result slot. ClassifyTerminalCompletionCorrelation identifies the exact checkpoint shape; ResolveCheckpointCompletionResult emits authority bound to its session, run, owner input, candidate digest, complete recipient set and digest, and checkpoint requirement. Every recipient must be consumed by that exact run. A same-boundary Steer batch can have multiple recipients while retaining one canonical owner. Neither command replaces a newer active run or changes its terminal facts. The boundary is the committed receipt’s boundary, not merely the input’s requested steer mode. Live realization records it with RecordBoundarySeq; cold recovery rehydrates it through RecoverInputCompletionBoundary only after checking the existing receipt’s run, sequence, and contributors against the consumed input. This uses the existing durable format. An idle steer request can execute a full turn at either RunStart or RunCheckpoint. Its RunResult remains an ordinary run completion: the apply boundary is not a terminal verdict. Only the exact consumed CompletedWithoutResult shape uses independent checkpoint completion authority. The runtime validates the complete durable owner/recipient batch before requesting that authority, then finalizes the genuine receipt through the existing atomic input-row compare-and-swap. A stale row, changed candidate, wrong owner/run, or inconsistent recipient set refuses without rewriting the evidence. Multiple checkpoint batches in one run remain distinguished by their canonical owner inputs. The owner-held candidate and its digest are the durable candidate authority. The boundary receipt does not independently attest the candidate variant; replacing both a candidate and its digest coherently before recovery cannot be detected as historical tampering from that receipt alone. Existing row fences still reject changes made after a completion witness was captured. Only a successful receipt commit retires the pending obligation. The runtime_pending_terminal_owners table is an index over those obligations, not an operator lock table. Clearing it, rewriting an input as abandoned, or fabricating a publication receipt is not a recovery operation. Ordinary run results and directed interaction publication retain their separate generated authority and exact EventStore receipt requirements.

Durable-Tail Recovery Ownership

A Session is domain state, not a portable proof of persistence. It carries the conversation, metadata, usage, and compact transcript-rewrite graph, but no embedded persistence authority. Physical currentness belongs to the store:
  • WholeBlob authority is the exact {session_id, store_revision, blob_sha256} issued with the serialized row.
  • HeadCanonical authority is the exact {session_id, store_revision, boundary_head, committed_head_token}. The small boundary head binds its message-row, rewrite, graph, component, and metadata prefixes.
The intra-turn persistence hook may write a physical successor before the runtime boundary transaction. Each successful write returns an explicit RunCheckpointReceipt naming the store profile, committed base, run, exact candidate identity, and a contiguous candidate_sequence. A later write in the same run must present the preceding receipt; a retry must match it exactly. The final boundary promotes the latest exact receipt rather than serializing or applying the accumulated state again. A shutdown race can still leave the latest candidate durable while its runtime boundary is uncommitted. Durable is not the same as runtime-committed, and recovery of that tail is machine-owned end to end. Ownership splits three ways:
  • The runtime store retains; SessionDocumentMachine classifies. Store-issued committed authority and provisional-tail authority decide what a reader may be served; a recovery candidate is never returned as an ordinary session. ClassifyDurableTail consumes the mechanically extracted tail shape (head relation, run-id cardinality, terminal stop reason, dangling/orphan tool counts) and emits DurableTailClassified with a DurableTailRecoveryClass: CompletedCandidate, InterruptedRepairableCandidate, or Ambiguous. The classification transitions are total and disjoint over the observation. A tail carrying any dangling tool call classifies Ambiguous: the call proves intent, not execution, so the tail is held for reconciliation rather than repaired. Only a store-proven descendant of the committed authority can classify as recoverable, and the relation comes from the store-loaded recovery source, never from a caller. VerifiedStrictDescendant: the physical head extends the authority’s exact message prefix. VerifiedRewriteDescendant: a compaction rewrote the transcript after the authority, and the store proves the descent from its persisted rewrite rows (they bridge both rewrite prefixes, and the authority’s exact row prefix chains into the first rewrite after it). Both relations pass the same shape checks, and neither stands in for the other: a descent the store cannot prove reaches the classifier as Unverifiable and classifies Ambiguous. Rewrite rows carry no run identity, so a rewrite descendant is observed from its last rewrite onward; when that live tail holds no message of the run it has no run id and classifies Ambiguous. The candidate id binds the exact store-issued base and candidate identities, run id, and candidate sequence, so classifying one physical tail can never authorize mutating a later one.
  • MeerkatMachine authorizes. AuthorizeDurableTailRecovery consumes the exact classified candidate together with typed projections of the durable evidence: the persisted machine-lifecycle row and its current-run fact (DurableRecoveryObservedLifecycle, DurableRecoveryObservedRun — a cold recovery drives a freshly registered authority whose in-process facts are vacuously quiescent, so the persisted row is the real evidence), the highest durably committed receipt compared against the candidate’s own content (DurableRecoveryPriorCommit — a candidate the receipts already cover refuses instead of committing a phantom duplicate boundary), and the attributability of the input rows the commit would terminalize (DurableRecoveryInputEvidence). Recovery is admissible only when both the in-process and the persisted facts are quiescent (Idle, Retired, or a missing row) with no current run and no recorded terminal for the candidate’s run; every other shape refuses. The one exception is the drive’s store-proven ownership (DurableRecoveryOwnership): when typed store facts prove that the drive holds the session’s exclusive write authority (on the built-in SQLite realm the session’s cross-process hosting claim; on a host store the realm fence this process holds after its open waited out any superseded lease), an earlier incarnation’s non-quiescent row, or its current-run fact naming the candidate’s run, no longer gates. A row naming another run, or one that does not decode, still refuses; the proof is never inferred from timing, and it covers earlier holders only (a drive offered another runtime owner’s claim, or whose own runtime owner has the session registered, proves nothing). A drive that hosts the session (ExclusiveHost) then decides as a restart does; a store-only writer (ExclusiveWriter) cannot run the run, so it takes the retired cuts and never continues it. The bounded TLC audit specs/machines/meerkat_machine/durable_tail_recovery_ownership_audit.sh checks the partition of the arms and these rules over the whole observation product. Commit verdicts arrive as DurableTailRecoveryCommitAuthorized (CommitCompleted, RepairAndCommitInterrupted, or CommitCompletedRetainInputs), carry the machine-minted boundary sequence — one past the last committed receipt for the run — and record the candidate run as the turn terminal. Hold and refusal verdicts arrive as DurableTailRecoveryAuthorized (HoldIntact, RefuseRecovery) and mutate nothing. CommitCompletedRetainInputs lets a completed candidate commit while retaining a separate unbound content input for ordinary redelivery. Only inputs durably proven bound to the recovered run are terminalized, including under this disposition. Unbound content inputs still hold interrupted candidates; unfenceable input evidence holds regardless of completed/interrupted shape. The observation decides which rows are bound: beside a run cursor the classifier judges to agree with the tail, the rows the cursor binds and those a committed receipt names, and a content row the cursor does not bind counts as unbound only when it names another run (one naming no run, or the candidate’s run, is not in the tail and stays queued); without an agreeing cursor, every content row not bound to the candidate run counts. These holds and the ambiguous-classification hold are machine-minted: no shell predicate may downgrade a commit authorization to a hold. Authorization never changes the lifecycle phase.
  • RuntimeStore realizes. Only the machine’s commit-authorizing effect may drive the recovery commit. Recovery seals a profile-aware PreparedRuntimeSessionCommit and submits it through RuntimeStore::commit_prepared_session_boundary as one boundary: the exact store candidate is promoted, the run-boundary receipt, input lifecycle transitions, terminal outcome, catalog projection, and outbox rows commit together. WholeBlob promotion reuses the candidate bytes already written; HeadCanonical promotion advances the committed authority to the exact physical head. Both are fenced on the observed store, lifecycle-row, and input-row identities, so a concurrent writer fails the whole boundary typed instead of being overwritten. No shell promotes or discards the tail. recover_durable_tail(&dyn RuntimeStore, &SessionId) is the only public preparation seam. The store loads an opaque PreparedDurableTailRecoverySource; recovery derives the classification, receipt facts, candidate identity, and physical CAS before sealing crate-owned PreparedRecoveryEvidence. There is no public request constructor or caller-mintable recovery capability.
The recovery rule is never-discard: every store-proven durable descendant is preserved. Recovery commits a completed tail as a recovered run boundary, closes an interrupted tail as interrupted (content preserved, a typed recovery notice appended, the original run terminalized — never requeued), or holds ambiguous evidence intact with autonomous execution blocked. Current stores expose only store-issued committed or provisional-tail authority; the former embedded-checkpoint projection-conflict vocabulary is not part of the live authority path.

Canonical Compositions

The canonical registry is canonical_composition_schemas() in crates/meerkat-machine-schema/src/catalog/mod.rs. These compositions are not marketing concepts or public APIs. They are contributor-facing guardrails for the runtime implementation.

Generated Artifacts

Generated and checked artifacts live in:

Contribution Rules

When a change affects lifecycle, routing, admission, credential state, mob membership, or scheduling, treat it as a machine-authority change until proven otherwise. Use this checklist:
  1. Identify the semantic owner.
  2. Add or update the catalog DSL if the legal states or transitions changed.
  3. Regenerate machine artifacts.
  4. Update production bridge code to call the generated authority path.
  5. Run the machine verification gates.
Do not add a side map, status enum, or handwritten reducer that decides the same fact in parallel with a machine.

Validation

Use the Make surface:
CI runs the relevant gates for normal development, and node scripts/affected-tests.mjs prints the regeneration, test and TLC commands a change needs locally. Use the direct targets when you are touching the catalog, generated kernels, composition routes, or runtime bridge code.

See Also