> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rkat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Stopped-Phase Revival

> Root-cause design record for the 0.7.19–0.7.23 resume-strand class: machine-owned revival of stopped sessions and cold-load snapshot reconciliation

# Stopped-Phase Revival — implementation plan (0.7.24)

Root cause (verified, wf\_63cf2cb3-27c + field forensics): the MeerkatMachine session
DSL classifies Stopped as terminal (only exits: UnregisterSessionStopped→Idle teardown,
Destroy→Destroyed), yet both resume seeds deliver registered sessions IN Stopped
(warm: RuntimeExecutorExitedFrom\*→Stopped, entry stays registered; cold:
RecoverRuntimeAuthorityStopped Initializing→Stopped from durable snapshot). Every
registration input accepts-and-preserves Stopped (RegisterSessionIdempotent,
PrepareBindingsStopped, EnsureSessionWithExecutorStopped self-loops), so the resume
build proceeds until the first Stopped-intolerant input — field: PublishLocalEndpoint
(per\_phase \[Idle, Attached, Running], dsl \~20194) via factory.rs:4628
install\_peer\_comms\_on → wedge, background repair retries forever. Five releases of
symptoms are this one missing re-admission arc plus ad-hoc per-input tolerance arms
(HydrateSessionLlmStateStopped comment at dsl:6417-6424 documents the whack-a-mole).

## Fix: machine-owned revival at the two intent-to-use seams

DSL edits in meerkat-machine-schema/src/catalog/dsl/meerkat\_machine.rs AND its mirror
meerkat-runtime/src/meerkat\_machine/dsl.rs (machine-check-drift enforces parity):

1. RegisterSession (\~5611) / RegisterSessionIdempotent (\~5639): remove Stopped from
   per\_phase. Add:
   * RegisterSessionResumesStopped: guard lifecycle\_phase==Stopped, same\_session
     (self.session\_id == Some(session\_id)), not\_draining (registration\_phase !=
     Draining). update: registration\_phase=Queuing; runtime\_stop\_deferred=false.
     PRESERVES session\_id, active\_runtime\_id/fence/generation/epoch, hydrated LLM
     identity/capability surface. to Idle. emit a `RuntimeNotice` with kind `Recover`
     and detail "stopped session re-admitted for resume". Semantics: a stopped executor is a fact about
     the previous epoch; a re-registered session with no executor is Idle.
   * RegisterSessionNewBindingFromStopped: guards new\_session\_binding + not\_draining;
     update = RegisterSession's full LLM-state reset block AND clear binding fields
     (active\_runtime\_id/fence/generation/epoch = None — new tenant, previous
     tenant's binding facts must not leak; verifier amendment). to Idle.
2. EnsureSessionWithExecutorStopped (\~9357): empty self-loop → revival mirroring the
   Idle arm: not\_draining guard; update registration\_phase=Active,
   runtime\_stop\_deferred=false; to Attached; emit a `RuntimeNotice` with kind `Recover`.
   (Fixes schedule delivery + mob re-dispatch with NO shell change — the executor
   claim at mod.rs:1101-1122 then grants.)
3. DELETE PrepareBindingsStopped (\~7546); remove Stopped from PrepareBindingsIdempotent
   per\_phase (\~7409). Post-revival PrepareBindings never legitimately sees Stopped;
   reaching it is a loud GuardRejected (ordering bug), never a silent rebind.
4. DELETE tolerance arms whose purpose was the missing arc: HydrateSessionLlmStateStopped
   (\~6425), PublishCommittedVisibleSetStopped (\~8252), SetSilentIntentsStopped (\~9395);
   remove Stopped from SetModelRoutingBaseline (\~6756), StagePersistentFilter (\~7361),
   RequestDeferredTools (\~7370) admissions. KEEP all Retired arms (Retired resume-build
   is the archive-completion flow, different invariant, out of scope).
   AMENDMENT-5 caution: stage\_persistent\_filter / request\_deferred\_tools /
   publish\_committed\_visible\_set are public MeerkatMachine APIs — sweep callers; each
   caller must either be post-revival or handle typed rejection.
5. AMENDMENT-1 (load-bearing): add RetireRequestedFromStopped (Stopped→Retired,
   mirroring RetireRequestedFromIdle guard family \~8315) and DELETE the shell phase
   probes / early-return matches in meerkat-mob/src/runtime/session\_service.rs
   (:93-104, :116-128). Registered-Stopped retire currently hits
   RuntimeControlPlane::retire at :105 → Retire has no Stopped arm → guard-reject.
   Sequence with provisioner.rs:950: add Stopped to the ask-21d disposal match ONLY
   after the Retire arm lands.
6. AMENDMENT-2: revival must persist durable lifecycle via machine-emitted typed
   effect (pattern: durability\_authority.action == DeleteSnapshot keying the unregister
   persist, session\_management.rs:1667-1675). stage\_session\_dsl\_input DISCARDS
   committed effects (dsl\_effects.rs:90-99) — route revival effects to the persistence
   channel; persist\_current\_machine\_lifecycle("resume") after revival commit.
   Without it, a revived-but-not-yet-turned session leaves durable Stopped visible
   cross-process (the ask-21d split-state read class).

## Class tests (can't-happen-again battery)

* TOTALITY SWEEP (class-killer): walk canonical\_machine\_schemas() MeerkatMachine
  transitions; assert the set of inputs admissible in phase Stopped == explicit
  allowlist DERIVED FROM THE DSL (not asserted): registration/revival inputs, ensure,
  BeginUnregisterSessionRetainsSnapshot, drain-feedback trio (conditional on
  Draining), UnregisterSession, Destroy, Retire (new), ResolveRuntimeOpsLifecycle
  Durability, observability self-loops (PublishEvent, ResolveRuntimeCompletionResult\*,
  SetPeerIngressContext, Abort/Wait...). NOTE: StopRuntimeExecutor has NO Stopped arm
  today — must NOT appear in allowlist (verifier: sweep red on day one otherwise).
* MACHINE UNITS: revival preserves identity/bindings/LLM state, registration==Queuing;
  ensure revives to Attached+Active; revival refused while Draining (BOTH entry
  orders: Begin at Stopped; Begin at Attached then executor exit → Stopped+Draining);
  new-binding-from-Stopped resets LLM state AND bindings; UnregisterSessionStopped
  still clears; retire-from-Stopped → Retired.
* RUNTIME WARM: prepare\_bindings\_after\_stop\_yields\_live\_machine\_and\_installs\_peer\_comms
  (THE field repro: register+ensure+stop via stop\_runtime\_executor, prepare\_bindings,
  assert phase != Stopped, install\_generated\_peer\_comms\_on\_target succeeds). Assert
  visible\_phase != Stopped after revival (projection sync, dsl:10573-10586).
  ensure\_after\_stop\_attaches\_new\_executor\_and\_admits\_input (schedule/mob re-dispatch
  class: accept\_input\_with\_completion admits, turn completes).
* RUNTIME COLD: durable Stopped snapshot (stop+unregister retains snapshot) → fresh
  machine over same store → prepare\_bindings → live phase, epoch cursor continuity,
  full comms build succeeds. Variant: seed snapshot verbatim (old-binary pattern).
* RACE PIN: StopRuntimeExecutor landing between revival and comms install re-enters
  Stopped and fails the build with the SAME "guard rejected ... PublishLocalEndpoint"
  string — correct fail-closed; pin so the old signature is distinguishable from
  regression.
* MOB CLASS: respawn/refresh-resume of stopped member (incl. comms wiring
  AddDirectPeerEndpoint post-revival); archive\_of\_stopped\_member\_durably\_retires;
  mob-seam routed PrepareBindings against Stopped member → typed rejection surfaced
  (composition.rs:346-358 route has no revival — production safe only because
  prepare\_local\_session\_bindings stages RegisterSession first; pin it).
* STOP-WINDOW GUARDS: stop→cleanup→unregister still retains snapshot from Stopped;
  register\_during\_drain\_rejects\_typed then succeeds after drain.

Red-verification: write tests + fix; then temporarily restore RegisterSessionIdempotent
per\_phase Stopped + empty EnsureSessionWithExecutorStopped and confirm the warm
runtime test deadlock/fails with the field error string.

## Codegen cascade

make machine-codegen (kernels, specs/machines/meerkat\_machine/model.tla, mob-seam
composition artifacts) + cargo xtask protocol-codegen (ulimit -s 65520 on macOS).
Gates: machine-check-drift, machine-authority-docs-gate, verify-machine-poster-coverage,
audit-generated-headers, runtime-authority-bypass, rmat-audit. TLC step bounds may need
adjustment (authority-lane precedent). No contracts wire-type changes.

## Bug B (historical): stale runtime snapshot vs store head — SUPERSEDED

meerkat-session/src/persistent.rs load\_authoritative\_session\_base\_with\_replay\_info
preferred the runtime snapshot whenever present, no freshness comparison; quarantine
only covered snapshot-absent. Field: snapshot froze at 83 msgs, store head 91; resume
loaded 83, save rejected by the append-only guard — permanent wedge.

The historical fix used a `SessionDocumentMachine` read-source input with
three typed observations —

1. store\_head\_extends\_snapshot: the head strictly extends the snapshot (prefix
   transcript digest equality, the save guard's own continuity proof);
2. store\_head\_has\_atomic\_physical\_ownership: the store can prove the head row
   is its current authenticated physical authority rather than an
   uncommitted projection;
3. session\_is\_live: a live session's snapshot lag is transient (the live runtime
   recommits past it); only a COLD load defers to the head.
   That embedded-checkpoint read-source vocabulary was removed in 0.8.11. Current
   stores issue committed and provisional-tail authority directly; recovery
   classifies a store-bound candidate and never treats a deserialized Session
   checkpoint stamp as authority.

Pins: test\_stale\_prefix\_runtime\_snapshot\_defers\_to\_extending\_store\_head (red-verified
against the old unconditional snapshot preference),
test\_unowned\_ahead\_store\_head\_stays\_snapshot\_served,
test\_diverged\_runtime\_snapshot\_stays\_authoritative,
test\_authoritative\_load\_ignores\_newer\_raw\_store\_projection\_when\_runtime\_exists (live),
and the rewritten cold tail of
test\_live\_runtime\_list\_status\_and\_resume\_fail\_closed\_on\_stale\_raw\_store\_metadata —
whose former expectation (resume failing closed against the longer persisted row)
WAS the field wedge, now asserted to resume successfully.