Coordination Boundaries
- WorkGraph is the durable shared-work ledger. It owns dependency readiness, work claims and leases, evidence, and terminal work verdicts.
- Schedule is the time authority. It owns trigger calculation, recurrence, occurrence claims, delivery attempts, and occurrence outcomes. A scheduled wake does not decide whether WorkGraph work remains ready or complete.
- Detached Jobs own durable background execution, while Runtime Delivery owns delivery identity, sequence, replay, and application progress. A job terminal receipt does not by itself prove that the target runtime applied the result.
- Peer delivery accelerates discovery of durable work and wakes disposable execution shells. A peer send or handoff receipt is not a WorkGraph claim, Schedule occurrence verdict, or proof that shared work completed.
- Durable claims are authorized only by their owning machine and store contract. WorkGraph authorizes work claims, Schedule authorizes occurrence claims, and peer FIFO removal requires the exact runtime admission receipt bound to its claim. Queue snapshots, counters, ages, correlations, and projections are observability facts and cannot authorize mutation.
Canonical Machines
The canonical registry iscanonical_machine_schemas() in
crates/meerkat-machine-schema/src/catalog/mod.rs.
MeerkatMachine and MobMachine are the two runtime kernels. The other
canonical machines protect narrower domain facts: auth, approval and durable
decisions, schedules and occurrences, detached jobs and runtime delivery,
session document and turn admission, WorkGraph lifecycle, attention, and
execution, scoped tool grants, and the writer fence of a host store realm. WorkGraph is an
optional subsystem authority for durable agent commitments, dependency-aware
claim state, and goal attention bindings. WorkExecutionLifecycleMachine owns
the cross-system obligation for one execution attempt without absorbing either
WorkGraph item truth or Mob Flow run truth. Its effects are realized
mechanically by the Mob composition facade, and every launch, evidence, or
closure outcome returns through a typed machine input before the binding
advances.
Two scoped authorities sit outside the canonical registry.
SessionPersistenceVersionAuthority owns admission of released session
envelope and metadata schema versions. Its catalog DSL lives at
crates/meerkat-machine-schema/src/catalog/dsl/session_persistence_version_authority.rs,
and its production expansion lives at
crates/meerkat-core/src/generated/session_persistence_version_authority.rs. It is a
small restore-time authority, not a persistence-currentness or recovery
authority, and it has no entry in canonical_machine_schemas().
MobHostBindingAuthority (catalog DSL source
crates/meerkat-machine-schema/src/catalog/dsl/mob_host_binding_authority.rs,
production expansion crates/meerkat-mob/src/machines/mob_host_binding_authority.rs).
It is the member host’s process-scoped, mob-keyed admission and dedup
authority for host-addressed mob commands: supervisor bind/rebind/revoke,
host-command admission, materialize admission/preflight with success-only
dedup memory, release admission with recorded-disposal replay, and the remote
turn-outcome journal. It follows the session_persistence_version_authority
precedent — one shared DSL body expanded into both crates, pinned by a
dedicated production-schema parity test
(crates/meerkat-mob/tests/mob_host_binding_authority.rs) and by kernel tests
instead of TLC — so it carries seam-inventory dispositions like any machine
but has no poster, no generated kernel, and no entry in
canonical_machine_schemas().
Principal control-scope grants (multi-host mobs §8) follow the
ToolExecutionPolicy split. The MobMachine owns the grant lifecycle — the
operator_grant_scopes / operator_grant_expiries facts, the
GrantOperatorScopes / RevokeOperatorScopes transitions, and the in-machine
revalidation of caller-proposed revoke partitions — while scope resolution
lives shell-side in the sealed ResolvedControlPolicy
(crates/meerkat-mob/src/control_policy.rs): private shape, one resolve() mint
path, no serde, fail-closed to the empty scope set. Expiry is data: the
machine never reads a clock; the enforcement chokepoint reads the wall clock
once per decision and passes now_ms in as a parameter, so a restored mob’s
expired grants stay expired with zero persisted derived state. Grant
durability is a runtime-metadata record (MobOperatorGrantRecord) written
only under GrantRecorded/GrantRevoked transition witnesses and replayed
on resume through the machine’s own GrantOperatorScopes input — never a
second enforcement source.
Identity entries (a host’s own durable identities, as opposed to worker
members; see Mob Architecture)
are MobMachine facts too. The machine owns which members are entries
(identity_entry_origins) and which open spawn ladders act for one
(spawn_exec_entry_custody): a BeginSpawnExec that carries entry custody
records the entry in the membership commit or the dormant seat that closes its
ladder, in the same step that seats the member. The legacy adoption
(AdoptLegacyIdentityEntry) decides from the shell’s raw observation of the
member’s spawn row, and the shell journals an adoption before it commits. The
replay of either row records the entry again (RecoverIdentityEntry, before
the boot plan is sealed). The admission probe answers an entry’s name without
custody IdentityMemberRejected, the worker reconcile helpers range over
workers only, and the agent lane’s member admission
(ResolveAgentLaneMemberAdmission) is a pure verdict the actor’s command
admission realizes. The handle’s entry reads project the published machine
state. A respawn of an entry’s member is a same-session rotation (below): the
identity stays seated and an entry from its retirement’s start to its next
incarnation’s seat, each one machine step, so a respawned entry no longer
passes through a window in which it is neither seated nor an entry. The
machine owns the respawn’s route: its fresh-life respawn arm (RespawnRunning,
whose input Respawn names the identity) admits no entry
(not_an_identity_entry), and its refusal twin
(RespawnIdentityEntryRefused, in Running, Stopped and Completed) answers
RespawnRefused with the reason IdentityEntryKeepsSession, so no
fresh-life step runs for an entry in any phase an entry lives in. The
fresh-life respawn’s committed archive (the topology-preserving
ObserveMemberRetirementArchived arms) refuses an entry too, whatever the
shell previewed. The actor previews the verdict and routes on its reason: an
entry’s respawn is that rotation, a successor spec for one is answered typed,
and the retire-then-spawn path previews it again before it retires anything
(its successor never takes entry custody). One shell step still decides from
the machine’s entry facts until the machine owns it: a seat with entry
custody of a name a worker holds is refused MemberAlreadyExists over the
published facts. No other shell code records, removes or derives an entry.
The ci profile has no identities, so the hand-written audit
specs/machines/mob_machine/identity_entry_audit.sh drives the entry arms in
its runs: one records, refuses and replays entries, one takes an entry
through the dormant seat, the live retirement, the rotation, the fresh-life
respawn’s refusal (in Running and Stopped) and the destroy, and the others
drive the declared apply below.
The declared apply (MobHandle::apply_declared_roster; see
Mob Architecture) is
MobMachine’s too. DeclareIdentityEntry answers one verdict per declared
identity (IdentityEntryVerdict), one arm per verdict, each guarded by the
pure helper identity_entry_verdict over the declaration and the member’s
facts, and each guarded to act only on the entries of the custody it
declares, by an explicit check: a roster’s declaration (created_custody
false) on an identity that is no entry or a Declared one, a store-owned
update’s (created_custody true, since 0.9.9) on its Created entry,
seated, owed its seat, hosted on its first use or awaiting its held
creation, and, answered RecordOnly (its row alone changes, and its
unarchive or settle seats the new spec), archived or awaiting its host’s
preparation; an entry of the other origin is taken by no arm. A store-owned
update is a declaration under created custody, so both custodies answer the
same verdict for the same facts, and MobMachine routes it from its own
step’s state. The declaration carries the shell’s raw
observations: whether the host’s rule requires the identity’s preparation
before its first seat (preparation_required, answered PreparationPending
for a roster-owned identity never seated), the identity
store’s intent rows, the session service’s durability, the role the session
an unseated identity’s seat would resume was last built under
(resume_session_role, so a seat under another role takes the migration
declared from that role or is held), and the kind of the host’s first-boot
request (first_boot), whose preparation the declaration’s own step ends
when it carries the request out. Its facts are the unseated entries with
their typed cause (identity_entry_unseated: SeatOwed,
HostedOnFirstUse, HeldCreation, PreparationPending, Archived), the
role hold
(member_host_preparation_causes, cause RoleMigrationUndeclared, which
refuses every wake as a first-boot preparation does, holds a Live local
member’s run starts through MeerkatMachine’s HostPreparation hold reason
and refuses the explicit resume of the member (answered held while the
mob is still Stopped, before the resume prepares any member, and again at
its rebuild), and which ends at a verdict
that restores the role or in the step that commits the declared one, never
at the verdict that declares the move, and with its entry wherever the
entry ends, role_hold_belongs_to_an_entry) and the archived session of an
entry’s retired member (identity_entry_archived_sessions). None of them has a
journal row of its own: the replay derives an entry’s unseated standing and
its archived session from its member’s retirement rows (the terminal row
records whether the retirement archived the session, session_archived, and
the replay passes whether the incarnation was hosted elsewhere,
hosted_incarnation), and the next apply declares the rest again. Some
row fields are journal-only facts, for the member lifecycle journal read
(MobHandle::member_lifecycle_events), and the replay reads none of them:
a retirement’s session, the entry it was the member of, its cause (the
seal’s mark) and the rotation whose terminal step retired it, and a seat’s
session and the rotation that seated it (a rotation’s terminal step, or
rotate_member’s fresh life of an identity no member holds, which follows
the generation the machine retains for the identity). The shell writes each
from the machine’s state before the step or from the step’s journal effect;
no machine input or effect field carries them. The seal
of a complete roster (SealDeclaredRoster) marks the seated Declared entries
it does not list (identity_entry_retiring) and removes the unseated ones,
and leaves an entry whose spawn ladder is open to a later seal
(marked_entry_has_no_spawn_in_flight), so no mark lands between a spawn
rollback’s start row and its Retire step; a declaration that lists a marked
entry again before its retirement started drops the mark. The shell retires
each marked member, its start row records the cause, and its replay passes
every such row
(RecoverIdentityEntryRetiring, which marks only the identity’s seated
Declared incarnation and takes a no-op arm for any other row). The end of an
entry’s retirement records the session it archived, and ends the entry
when the seal, a one-entry unlist or a removal marked it (a removal’s end
also tombstones the name); an archive’s mark leaves the entry unseated,
Archived; any other end keeps the entry unseated, owed its seat (hosted
on its first use when placed or externally bound). The
dormant seat passes its raw observation that the session is archived and
revivable, and its guard binds an archived session only when it is the
entry’s record; it takes a fresh session only once the record is let go
(ForgetArchivedSession: the session was observed absent, or the host asked
for a fresh life), and it never seats an entry hosted on its first use or
awaiting its held creation. A seat ends a first-boot preparation only: a
role hold survives it. A placed spawn holds an entry’s custody only for an
entry hosted on its first use
(placed_entry_custody_is_hosted_on_first_use). Applies are serialized by
the apply-open fact (declared_apply_open): OpenDeclaredApply answers
Opened, or Busy while an apply is open, and CloseDeclaredApply closes
it in every phase. The shell closes it through the lease the opening step
made, on the internal lane: no grant and no Shutdown step refuses the close.
A mob reset ends every identity entry, and it commits
only on an empty roster (ResetToRunning’s roster_empty guard): its
preflight, AdmitReset, answers the reset’s other guards, changing nothing,
before the retirement batch that empties the roster. It also ends every
identity’s retained incarnation history, as a restart after the reset does
(the resume replays only the events after the last reset): it folds each
retained generation into the identity’s high-water
(identity_generation_highwater), so the next incarnation still mints above
every generation the identity used, and drops the retained generation,
fence, profile and runtime mode together. The generation high-water audit’s
reset run checks both, with a witness of the fold and one mutant per half. The audit’s apply runs
(apply, legacy, resume, roles, hold, held and placed) drive
the declare arms, the seal (which leaves a member whose spawn is in flight
unmarked), the legacy adoption before an unlisted retirement, the dormant
seat with and without an archived record, the retirement to its terminals,
the reset over an emptied roster, the role hold against the wakes of every
cause and the explicit resume, held creation, and a placed identity’s
custody, remote commit, retirement and replay, with witnesses and mutants.
Created identity entries (IdentityEntryOrigin::Created, since 0.9.9) are
entries of the same map, recovered from the identity store at a start. The
origin is the entry’s provenance and the custody of its desired record: it
decides the recovery, which writer persists a change of that record, the name
guards and the reservations below, never the member’s profile, labels, tools,
grants or wiring. Since 0.9.9 a created entry can be placed, externally
bound, held for its host’s creation or prepared by its host first, as a
declared one can: created custody takes a Created entry owed its seat,
hosted on its first use or held for its creation, and one awaiting its
preparation only for the spawn of a construction held for it, on a local
or a placed ladder (placed_entry_custody_is_hosted_on_first_use holds a
placed ladder of either custody to an entry hosted on its first use). After
the journal replay and before the boot plan is sealed, the shell sends one
RecoverCreatedIdentityEntry per created entry row, with the row’s presence
(RecoveredPresence: Present, PresentUnseated, PendingConstruction,
Tombstone, Archived, Unreadable) and the session the identity’s member
was last bound to in the replayed journal, and MobMachine decides from them
and its replayed facts, in Running or Stopped. A present row records a
seated local member a Created entry, or, when the replay left the identity
without a member, records the entry unseated (SeatOwed) with that session
as its revival record; a pending row records it unseated for its host’s
preparation (PreparationPending), and a pending row of a held construction
whose member the replay seats (its publication committed before a crash)
records the member a Created entry; an archived row records it unseated
(Archived); a removed or archived row whose member’s retirement the replay
holds in flight records the removal or the archive in flight
(identity_entry_removing, identity_entry_archiving), one mark per entry,
so a second row contradicting a recovered mark reserves the name as
unreadable; a removed row
records the name’s tombstone (identity_entry_tombstones); and an
unreadable row, or one the replay
contradicts (a tombstone of the name, or a spawn ladder open for it,
included), reserves the name
(identity_entry_unreadable). No arm records an entry for a tombstoned
name, and no replay seats a member under one, or under an archived entry.
The store holds an archive’s row beside its member’s roster row while the
archive’s retirement runs: the replay, which comes first, seats that member
and the scan then marks the archive in flight; a replay that came after the
scan recorded the entry archived is refused, so MobMachine does not depend
on that order. Nor does a Declared row’s replay replace a Created entry (a
store-owned identity never turns roster-owned): replayed first, a Declared
entry stands and the scan’s row of it reserves the name; scanned first, the
replay of a Declared row over the Created entry is refused. A Created row
may replay over a Declared origin: the journal records no end of an entry
that ends unseated (the seal’s or a one-entry unlist’s), so a later
store-owned incarnation’s row takes over the stale origin of a roster-owned
identity whose entry ended. MobMachine does rely on the boot’s order for a
row pending its host’s preparation: the start replays the journal
(seed_mob_authority_sync_from_events, then the placed carriers’
recover_current_committed_placed_carrier_signals) before it scans the
created entry rows (recover_created_identity_entries). Scanned first, a
pending row would record an identity without a member awaiting its
preparation, and a replay that seated the identity after it would drop that
cause: the start would neither settle the row of a published construction
nor reserve the name a member contradicts. The replay
reserves the name of a member row whose origin token this build does not
know the same way. A removed entry’s name is never an entry or a member
again; an unreadable name keeps what the replay recorded for it (a
Declared entry, a member) and takes nothing new: the admission probe
answers IdentityNameReserved (no spawn ladder opens, and every ladder
close, the membership commits, the remote commit and the dormant seat,
refuses a reserved name: a ladder opened before the reservation seats no
member, and the shell aborts it with the name’s admission answer,
MobError::IdentityMember), the worker reconcile helpers
skip it, the worker ensure refuses it (name_not_reserved on
EnsureMemberRunningMissing, which the shell answers with the name’s
admission answer), the declared apply refuses it typed (RefusedNameTombstoned,
RefusedNameUnreadable), the legacy adoption answers NotAdoptable, and
the agent lane’s admission answers the name’s member IdentityMember.
Custody of one origin never takes
an entry of the other (IdentityMemberOfAnotherOrigin): entry custody takes
a Declared entry’s name only, and created custody (BeginSpawnExec’s and
ProbeMemberAdmission’s created_custody, recorded in
spawn_exec_created_custody) a Created entry only, never both; the dormant
seat that closes a created ladder takes the entry’s revival record and keeps
the member a Created entry, and seats only an entry owed its seat. A
roster’s declaration never declares over a Created entry
(RefusedNameOwnedByCreated, which the shell answers as a typed note, a
quiet steady state), and the seal unlists readable Declared
entries only. A reset keeps the reserved names, which the identity store
holds, and drops created custody; the destroy clears both. The invariants
tombstone_is_never_an_entry, entry_custodies_are_disjoint and
archived_entry_is_store_owned hold these facts (the earlier invariant
unseated_created_entry_is_preparing_or_owed is retired: a store-owned
update’s declaration records a created entry hosted on its first use or
held, and the created identity audit’s scan run checks what the scan alone
records). The shell re-seats an entry owed its seat once the mob runs, as
the handle’s own command through that created custody, from the row’s spec
(or, for a row written before rows carried their spec, from its member’s last
spawn row with its roster labels), and MobMachine’s guards decide each seat.
Store-owned identity changes are MobMachine’s too (since 0.9.9; see
Mob Architecture).
Each is decided in its commit step, from the shell’s raw observations, before
the shell writes the change’s store transaction, and a refusal leaves the
change’s prepared authority unapplied: AdmitIdentityChange answers the
change’s admission (IdentityChangeAdmissionKind) from the store’s receipt of
the operation, the consent’s form, whether a decision’s permit binds the
change and admits a commit now, whether the requester is the host operator,
whether an agent requester’s grant is current at this final sink, whether the
spec the change was decided against is the identity’s (StaleSpec
otherwise) and the host’s changes hold (identity_changes_held, set by
SetIdentityChangesHeld); RecordCreatedIdentityEntry records a new
store-owned entry, unseated with the cause its helper derives from the
request’s facts (whether its host’s rule requires its preparation, its
placement, a held creation) in identity_entry_verdict’s precedence, under
a free name only (CreatedIdentityRecordVerdict; an unlisted roster-owned
identity’s archived session record holds its name); and
RemoveIdentityEntry, ArchiveIdentityEntry and UnarchiveIdentityEntry
answer one identity’s removal, archive and unarchive
(IdentityEntryChangeVerdict). A removal or an archive of a seated entry
marks its ending (identity_entry_removing, identity_entry_archiving) and
the end of its member’s retirement consumes the mark, tombstoning the name
or leaving the entry unseated, Archived, on its archived session; the
destroy’s retirement ends the mark with the entry. An unseated entry is
tombstoned at once, a held construction of it ending in the same step (an
accepted activation is in flight until its publication ends), and archived
at once unless it awaits its host’s preparation, whose archive is refused
(PreparationPending, the change’s typed refusal host_preparation_pending).
At most one ending mark lands on an entry, only on a seated one
(entry_ending_marks_are_seated_and_disjoint). UnlistIdentityEntry
unlists one Declared entry, decided on that identity’s own facts
(UnlistEntryVerdict): its seated member’s retirement marked as the seal
marks one, an entry no member holds ended at once, one whose member step
is in flight deferred, and nothing else read. Each answers one arm per
verdict through a pure helper, with a MobNotRunning arm per other phase.
MobMachine keeps the mob’s owned edges too (since 0.9.11): one custody map,
identity_edge_owners, from each edge an identity change declared to its
owner, the identity whose change declared it. DeclareOwnedIdentityEdge
declares an edge a store-owned identity’s change lists, to an identity of
either custody (OwnedEdgeDeclareVerdict: a second declaration of an owned
edge gives no custody, and a peer that is no identity is PeerUnknown);
EndOwnedIdentityEdge ends one by its owner’s change or by the host’s policy
reload that denies it (OwnedEdgeEndCause), naming its owner
(OwnedEdgeEndVerdict: another identity’s end is NotOwner); a removal ends
its identity’s edges, whichever end owns them, in its own step
(RemoveIdentityEntry), and the reset and the destroy end them all;
AdmitMemberUnwire refuses a plain unwire of an owned edge in force
(MemberUnwireAdmission); and the boot recovers each edge from its owner’s
row (RecoverOwnedIdentityEdge). The map is the edge’s desired record. An
edge is in force while both its ends are identity entries (the helper
owned_edge_in_force, and its complement owned_edge_out_of_force): only
then does the shell realize it
(wiring_edges) once both its ends are seated, each realization (a seat’s,
a spawn’s or a pass’s) reading MobMachine’s verdict before its wire
(AdmitOwnedEdgeRealization, OwnedEdgeRealization), and only then is it
refused a plain unwire. An archive of an end leaves the edge owned and
unrealized; an unlisting of an end leaves it owned and out of force, so a
worker that takes the end’s free name is no end of it, and the end’s
re-listing brings it back in force. The invariants
owned_edge_owner_is_a_store_owned_end and
owned_edge_joins_no_tombstone hold these facts. An import restores the
edges of the rows it restores as the boot recovers them, after the boot
plan’s seal too (RestoreOwnedIdentityEdge, OwnedEdgeRestoreVerdict), and
writes each restored row with only the edges it restored (an edge another
identity owns meanwhile is not the row’s): the custody map an import leaves
is the one the next start reads.
MobMachine decides an adoption into store custody too (since 0.9.11):
AdoptIdentityIntoStore takes a roster-owned identity (Roster: a Declared
entry seated on a local member bound to a session, or owed its seat) or a
worker member (Worker: seated, with no entry, local and bound to a session,
spawned with no profile or model override (spawn_overridden), at the
generation its host observed) into store custody as a Created entry,
its member’s runtime, generation and session untouched. A pure helper,
identity_adoption_verdict, decides over the name’s facts, one arm per
verdict (IdentityAdoptionVerdict): a member step in flight is
AdmissionUnsettled, an open declared apply DeclaredApplyOpen, a stale
generation PredecessorMismatch, and any other shape no adoption takes
NotAdoptable, with its reason (IdentityNotAdoptableReason, a worker’s
override among them: ProfileOverride); the shell
then declares the member’s current edges to identities owned edges of the
adopted identity in the same step.
MobMachine decides the downgrade cycle too (since 0.9.11).
PurgeIdentityRows admits the host operator’s purge of the mob’s rows of
schema version 2 and later only under held identity changes, and only when
the mob’s scan read any such row (rows_present; none is
NothingToPurge); the step’s
store transaction removes the rows when their digest is the export’s, and
the machine records only identity_rows_purged, which refuses every later
identity change (IdentityChangeAdmissionKind::RowsPurged) until the mob
restarts. A start that reads no row of a Created entry’s name
(RecoverRowlessCreatedEntry) reserves it, unreadable and rowless
(identity_entry_rowless), so nothing acts on it and no seat binds it; the
invariant created_origin_without_row_is_unreadable holds this.
ImportIdentityRecord decides each exported record, one arm per verdict
(IdentityImportVerdict): the record comes back for a member seated under
its name in place (a plain member only on the session the record names),
for an unseated name owed its seat on its archived
session, as recorded for a pending, an archived or a removed name, or by the
adoption’s step for a roster’s adopted identity whose roster declared it
again; a name another holder keeps is reported and left, and a rowless
entry’s name a member holds now is released to that member.
The host’s preparation before an identity’s first seat is MobMachine’s for
either custody. BeginPreparationConstruction holds an attempt of a
constructed preparation (identity_entry_constructions_held);
SettleHostPreparation settles a preparation without an attempt (the entry
is owed its seat; a constructed one is refused ConstructionRequired) or
accepts the activation of exactly the held attempt it names, decided before
the attempt’s deadline and binding its construction, moving it to
identity_entry_constructions_activating; and EndPreparationConstruction
ends an attempt by its abort, its deadline, a refused construction or a
failed publication, an activating one only by a failed publication. A spawn
ladder takes an entry awaiting its preparation only while a construction is
open for it, held or activating (the construction’s own spawn opens its
ladder at its publication, after the settle accepted its activation), and
the membership commit that publishes it requires its accepted activation and
ends the attempt. At most one attempt of an identity
is open (construction_is_held_or_activating).
The ci profile has no identities, so the hand-written audit
specs/machines/mob_machine/created_identity_audit.sh drives the created
entries’ arms in fourteen runs, in two parts. Its recovery part has four:
- The scan run scans a row of every presence, with and without a revival
record, in Running and Stopped, until the boot plan’s seal. The rows are
for a replayed seated member (with or without its
Declaredentry’s row, replayed before or after the scan; the member’s own replay never follows the scan of its row pending its host’s preparation, the boot’s order, which the names run keeps too) and for an identity without a member. The run reaches every outcome, the contradicted rows included, exactly one arm answers every row, and no step turns aCreatedentryDeclared. - The reseat run opens, aborts and seats spawn ladders with every pair of custodies over the scanned rows, alongside the worker ensure, the worker reconcile and the destroy. It checks that no ladder opens for a reserved name, that a ladder’s custody matches its name’s origin, that a seat under created custody binds the entry’s revival record, that a row of a name an open ladder acts for reserves it, and that no worker ensure records a reserved name for a spawn.
- The names run drives the declared apply, its seal, the legacy adoption,
the agent lane’s admission and the admission probe over
Created, removed and unreadable names. Exactly one arm answers each, and the seal never touches aCreatedentry. - The placed run scans the row of a member seated on a member host.
- The admit run drives every change’s admission from every fact, over the only facts it reads (the composition authority, the changes hold, a running and a stopped mob): exactly one arm answers each. The admission changes nothing, so no other run fires it.
- The commit run drives the record of a name from every request’s facts, the changes hold and a seat or membership commit under created custody: exactly one arm answers each record, a new entry takes only a free name (no entry of either custody, member, ladder, reservation or archived session record holds it), its cause follows the declared apply’s precedence over its facts, and an entry held for its creation is never seated Dormant.
- The update run declares the same change of a roster-owned and a store-owned identity, owed their seats or seated under the same spec: exactly one arm answers each, and both answer the same verdict for the same facts.
- The removal run removes a roster-owned identity through its retirement (or ends it by the destroy with its mark) and a store-owned one unseated, seated through its retirement, or archived: every removal’s end but the destroy’s tombstones its name.
- The archive run archives a seated store-owned identity through its
retirement and unarchives it onto its archived session: an archived
identity keeps its name and is never seated, an identity awaiting its
preparation is never archived, the update of either changes its row only
(
RecordOnly), and a roster-owned identity’s archive is refused. - The preparation run drives the start, settle and end of held
constructions of either custody with their ladders (the construction’s
own ladder opening after its activation was accepted included), seats,
membership commits, archives and removals, firing the attempts’ arms
that change state: an entry awaiting its preparation is seated only by
its accepted activation, no end of an accepted activation but its failed
publication is enabled, a
constructedpreparation is never settled without an attempt, and its update changes its row only. - The prepverdicts run drives every arm of an attempt’s start, settle and end from every fact, over the facts their verdicts read: exactly one arm answers each.
- The placement run seats a store-owned identity hosted on its first use by a placed or peer-only spawn holding its created custody.
- The unlist run unlists a roster-owned identity through its retirement beside a store-owned one: exactly one arm answers each unlist, which decides on its identity alone, and no record takes the name of an entry or of the unlisted identity’s archived session record.
- The recovered run replays a seated store-owned identity, its
Createdentry row included, and scans its one row, a removal’s or an archive’s (the store keeps one row per identity), before or after the journal replay. It carries the recovered mark’s retirement to its terminal, running or stopped: no replay seats a member under an archived entry, and the boot plan’s seal, a stop and a resume leave an archived identity archived.
MobMachine facts. A pending rotation records its
identity’s next session (member_rotation_session_ids), the endpoint it
seats (member_rotation_peer_endpoints) and its target
(member_rotation_targets): FreshSession (a reset, a deletion) seats the
next incarnation on a new session, SameSession (a respawn, a declared role
migration) on the member’s own. A same-session rotation starts the
incarnation’s retirement in its own step (BeginMemberRotation, target
SameSession) and keeps the binding; its retirement releases the runtime
registration and keeps the session (disposal KeptForSuccessor, which no
plain archive takes), and its terminal (the SameSession arms of
CommitMemberRotation) retires the old incarnation and seats the next one on
that session in one step, so the identity is never unseated. A releasing
retirement of a member whose same-session rotation stands is refused: its
re-drive preserves the binding. A same-session start takes the shell’s
observation of the kept session (session_archived), which its
session_not_archived guard refuses. The retirement’s start row journals the
rotation, and its replay records the rotation again
(RecoverMemberRotationIntent), so a mob restarted mid-rotation finishes it
with the rotation’s own terminal and no host record, while the plain archive
waits; an admitted destroy beats it. The start does not re-drive the
FreshSession rotation of a member its host marked host-pending: the host’s
re-issue finishes it. The start, the actor’s own re-drive and the re-drive of
a caller other than the host (redrive_retirement_unless_held_for_host)
decide that hold by one shell predicate over the machine’s facts, in the
actor’s own step: host-pending and a FreshSession target. A same-session
terminal keeps a host’s first-boot preparation
(member_host_preparation_causes, cause FirstBootRow): only the host’s seat
step ends it, so the successor of a host-pending member is seated Dormant and
every wake of it is refused until then. Under the member’s own role the
same-session terminal keeps the member’s endpoint and its wiring and external
peer edges, which the replay of its MemberRetired row ends, so its journal
batch records them again after the next incarnation’s row. Every same-session
start records the role it starts with (profile_name, held in
member_rotation_profile_names while the rotation stands, recorded again by
the replay and dropped with the rest of the intent), and both same-session
terminals refuse a grant of any other role (grant_takes_the_started_role)
and take the role change from it: the start’s retirement prepared the member’s
topology for that role. A same-session successor under another role runs under
that role’s comms name: the start records an endpoint only for another role,
under the member’s own peer id (endpoint_only_for_another_role,
endpoint_keeps_the_members_peer_id, and the same guards on the replay), the
shell’s retirement unwires the member’s peers’ trust as a plain retirement
does, and the terminal ends the member’s edges and installs the recorded
endpoint (or, with none recorded, owes one until the successor’s first wake
registers it). The invariants same_session_rotation_keeps_its_binding and
rotation_started_role_is_a_same_session_rotations hold the started role
exactly while a same-session rotation is pending. The
release verdict (ClassifyMemberRelease) takes the incarnation a scoped
release names and answers Stale for any other. The hand-written
specs/machines/mob_machine/member_residency_audit.sh drives these arms in
its rotate-same and recover-rotation runs, and replays a committed
same-session terminal’s rows, its kept edges among them, in its
replay-same run.
The retire of an identity that holds no roster incarnation (never a member,
or already retired) is a MobMachine answer too, in every phase. The handle
reads the roster only to choose the command: finding no incarnation, it asks
the actor for RetireAbsent, the input the actor’s own retirement submits
when it meets an absent member, and answers what the machine answers.
RetireAbsent has an arm in each phase (RetireAbsentRunning,
RetireAbsentStopped, RetireAbsentCompleted, RetireAbsentDestroyed, the
last for a destroyed mob whose actor still serves its destroy’s retry). Each
changes nothing while the identity maps to no runtime (identity_absent)
and refuses an identity the machine still maps, such as a successor whose
spawn committed after the caller read the roster. The handle answers that
refusal as the spawn still in progress
(MobError::LifecycleOperationPending, retryable). The input names no
incarnation, so it never retires or cancels a later incarnation of the
identity. The hand-written
specs/machines/mob_machine/member_lifecycle_audit.sh fires the four arms
and checks, in its scope run, that a RetireAbsent arm is enabled exactly
when the identity maps to no runtime.
Consumed Checkpoint Completion Recovery
A live steer checkpoint can durably consume an input before its completion receipt is finalized. ItsCompletedWithoutResult candidate is not a new
RunResult, and a pending completion does not make the input abandoned.
MeerkatMachine owns checkpoint correlation through the consumed input’s
existing run association, independently of the latest run-result slot.
ClassifyTerminalCompletionCorrelation identifies the exact checkpoint
shape; ResolveCheckpointCompletionResult emits authority bound to its
session, run, owner input, candidate digest, complete recipient set and digest,
and checkpoint requirement. Every recipient must be consumed by that exact
run. A same-boundary Steer batch can have multiple recipients while retaining
one canonical owner. Neither command replaces a newer active run or changes
its terminal facts.
The boundary is the committed receipt’s boundary, not merely the input’s
requested steer mode. Live realization records it with RecordBoundarySeq;
cold recovery rehydrates it through RecoverInputCompletionBoundary only
after checking the existing receipt’s run, sequence, and contributors against
the consumed input. This uses the existing durable format. An idle steer
request can execute a full turn at either RunStart or RunCheckpoint.
Its RunResult remains an ordinary run completion: the apply boundary is not
a terminal verdict. Only the exact consumed CompletedWithoutResult shape
uses independent checkpoint completion authority.
The runtime validates the complete durable owner/recipient batch before
requesting that authority, then finalizes the genuine receipt through the
existing atomic input-row compare-and-swap. A stale row, changed candidate,
wrong owner/run, or inconsistent recipient set refuses without rewriting
the evidence. Multiple checkpoint batches in one run remain distinguished
by their canonical owner inputs.
The owner-held candidate and its digest are the durable candidate authority.
The boundary receipt does not independently attest the candidate variant;
replacing both a candidate and its digest coherently before recovery cannot
be detected as historical tampering from that receipt alone. Existing row
fences still reject changes made after a completion witness was captured.
Only a successful receipt commit retires the pending obligation. The
runtime_pending_terminal_owners table is an index over those obligations,
not an operator lock table. Clearing it, rewriting an input as abandoned,
or fabricating a publication receipt is not a recovery operation. Ordinary
run results and directed interaction publication retain their separate
generated authority and exact EventStore receipt requirements.
Durable-Tail Recovery Ownership
ASession is domain state, not a portable proof of persistence. It carries
the conversation, metadata, usage, and compact transcript-rewrite graph, but
no embedded persistence authority. Physical currentness belongs to the store:
- WholeBlob authority is the exact
{session_id, store_revision, blob_sha256}issued with the serialized row. - HeadCanonical authority is the exact
{session_id, store_revision, boundary_head, committed_head_token}. The small boundary head binds its message-row, rewrite, graph, component, and metadata prefixes.
RunCheckpointReceipt naming the store profile, committed base, run, exact
candidate identity, and a contiguous candidate_sequence. A later write in
the same run must present the preceding receipt; a retry must match it exactly.
The final boundary promotes the latest exact receipt rather than serializing
or applying the accumulated state again.
A shutdown race can still leave the latest candidate durable while its
runtime boundary is uncommitted. Durable is not the same as
runtime-committed, and recovery of that tail is machine-owned end to end.
Ownership splits three ways:
- The runtime store retains;
SessionDocumentMachineclassifies. Store-issued committed authority and provisional-tail authority decide what a reader may be served; a recovery candidate is never returned as an ordinary session.ClassifyDurableTailconsumes the mechanically extracted tail shape (head relation, run-id cardinality, terminal stop reason, dangling/orphan tool counts) and emitsDurableTailClassifiedwith aDurableTailRecoveryClass:CompletedCandidate,InterruptedRepairableCandidate, orAmbiguous. The classification transitions are total and disjoint over the observation. A tail carrying any dangling tool call classifiesAmbiguous: the call proves intent, not execution, so the tail is held for reconciliation rather than repaired. Only a store-proven descendant of the committed authority can classify as recoverable, and the relation comes from the store-loaded recovery source, never from a caller.VerifiedStrictDescendant: the physical head extends the authority’s exact message prefix.VerifiedRewriteDescendant: a compaction rewrote the transcript after the authority, and the store proves the descent from its persisted rewrite rows (they bridge both rewrite prefixes, and the authority’s exact row prefix chains into the first rewrite after it). Both relations pass the same shape checks, and neither stands in for the other: a descent the store cannot prove reaches the classifier asUnverifiableand classifiesAmbiguous. Rewrite rows carry no run identity, so a rewrite descendant is observed from its last rewrite onward; when that live tail holds no message of the run it has no run id and classifiesAmbiguous. The candidate id binds the exact store-issued base and candidate identities, run id, and candidate sequence, so classifying one physical tail can never authorize mutating a later one. MeerkatMachineauthorizes.AuthorizeDurableTailRecoveryconsumes the exact classified candidate together with typed projections of the durable evidence: the persisted machine-lifecycle row and its current-run fact (DurableRecoveryObservedLifecycle,DurableRecoveryObservedRun— a cold recovery drives a freshly registered authority whose in-process facts are vacuously quiescent, so the persisted row is the real evidence), the highest durably committed receipt compared against the candidate’s own content (DurableRecoveryPriorCommit— a candidate the receipts already cover refuses instead of committing a phantom duplicate boundary), and the attributability of the input rows the commit would terminalize (DurableRecoveryInputEvidence). Recovery is admissible only when both the in-process and the persisted facts are quiescent (Idle,Retired, or a missing row) with no current run and no recorded terminal for the candidate’s run; every other shape refuses. The one exception is the drive’s store-proven ownership (DurableRecoveryOwnership): when typed store facts prove that the drive holds the session’s exclusive write authority (on the built-in SQLite realm the session’s cross-process hosting claim; on a host store the realm fence this process holds after its open waited out any superseded lease), an earlier incarnation’s non-quiescent row, or its current-run fact naming the candidate’s run, no longer gates. A row naming another run, or one that does not decode, still refuses; the proof is never inferred from timing, and it covers earlier holders only (a drive offered another runtime owner’s claim, or whose own runtime owner has the session registered, proves nothing). A drive that hosts the session (ExclusiveHost) then decides as a restart does; a store-only writer (ExclusiveWriter) cannot run the run, so it takes the retired cuts and never continues it. The bounded TLC auditspecs/machines/meerkat_machine/durable_tail_recovery_ownership_audit.shchecks the partition of the arms and these rules over the whole observation product. Commit verdicts arrive asDurableTailRecoveryCommitAuthorized(CommitCompleted,RepairAndCommitInterrupted, orCommitCompletedRetainInputs), carry the machine-minted boundary sequence — one past the last committed receipt for the run — and record the candidate run as the turn terminal. Hold and refusal verdicts arrive asDurableTailRecoveryAuthorized(HoldIntact,RefuseRecovery) and mutate nothing.CommitCompletedRetainInputslets a completed candidate commit while retaining a separate unbound content input for ordinary redelivery. Only inputs durably proven bound to the recovered run are terminalized, including under this disposition. Unbound content inputs still hold interrupted candidates; unfenceable input evidence holds regardless of completed/interrupted shape. The observation decides which rows are bound: beside a run cursor the classifier judges to agree with the tail, the rows the cursor binds and those a committed receipt names, and a content row the cursor does not bind counts as unbound only when it names another run (one naming no run, or the candidate’s run, is not in the tail and stays queued); without an agreeing cursor, every content row not bound to the candidate run counts. These holds and the ambiguous-classification hold are machine-minted: no shell predicate may downgrade a commit authorization to a hold. Authorization never changes the lifecycle phase.RuntimeStorerealizes. Only the machine’s commit-authorizing effect may drive the recovery commit. Recovery seals a profile-awarePreparedRuntimeSessionCommitand submits it throughRuntimeStore::commit_prepared_session_boundaryas one boundary: the exact store candidate is promoted, the run-boundary receipt, input lifecycle transitions, terminal outcome, catalog projection, and outbox rows commit together. WholeBlob promotion reuses the candidate bytes already written; HeadCanonical promotion advances the committed authority to the exact physical head. Both are fenced on the observed store, lifecycle-row, and input-row identities, so a concurrent writer fails the whole boundary typed instead of being overwritten. No shell promotes or discards the tail.recover_durable_tail(&dyn RuntimeStore, &SessionId)is the only public preparation seam. The store loads an opaquePreparedDurableTailRecoverySource; recovery derives the classification, receipt facts, candidate identity, and physical CAS before sealing crate-ownedPreparedRecoveryEvidence. There is no public request constructor or caller-mintable recovery capability.
Canonical Compositions
The canonical registry iscanonical_composition_schemas() in
crates/meerkat-machine-schema/src/catalog/mod.rs.
These compositions are not marketing concepts or public APIs. They are
contributor-facing guardrails for the runtime implementation.
Generated Artifacts
Generated and checked artifacts live in:Contribution Rules
When a change affects lifecycle, routing, admission, credential state, mob membership, or scheduling, treat it as a machine-authority change until proven otherwise. Use this checklist:- Identify the semantic owner.
- Add or update the catalog DSL if the legal states or transitions changed.
- Regenerate machine artifacts.
- Update production bridge code to call the generated authority path.
- Run the machine verification gates.
Validation
Use the Make surface:node scripts/affected-tests.mjs prints the regeneration, test and TLC
commands a change needs locally. Use the direct targets when you are touching
the catalog, generated kernels, composition routes, or runtime bridge code.
