Skip to main content
This page is the canonical settled-state contract for session and runtime behavior.

Ownership model

Meerkat has one canonical semantic path:
  • the runtime control plane (MeerkatMachine in meerkat-runtime) owns keep_alive, Queue/Steer routing, comms drain lifecycle, external-event admission, and request/turn commit semantics; runtime-backed surfaces reach those semantics only by lowering into it
  • SessionService is the substrate lifecycle seam used by those surfaces
  • EphemeralSessionService supplies in-memory session storage and execution for browser sessions behind MeerkatMachine; direct use through build_ephemeral_service remains available for explicit standalone tests and embeddings
The runtime-backed path is:
The runtime-backed build contract has an explicit binding seam:
  • runtime-backed surfaces should call MeerkatMachine::prepare_bindings(session_id)
  • those bindings flow into SessionBuildOptions.runtime_build_mode = RuntimeBuildMode::SessionOwned(bindings)
  • standalone / embedded / test-only builds should opt into RuntimeBuildMode::StandaloneEphemeral explicitly instead of relying on silent fallback
Standalone sessions retain one generated authority across turns. After a coherent terminal effect, CloseStandaloneTurn releases the epochless, unplaced session’s lifecycle while preserving its terminal evidence and model/tool-routing state. It does not mint a durable commit receipt or close a runtime-owned session; SessionOwned terminal publication remains with the runtime commit owner. SessionRuntimeBindings is the epoch-local runtime handle for a session. It carries:
  • session_id
  • epoch_id
  • the canonical OpsLifecycleRegistry
  • shared consumer cursor state used for recovery-safe completion visibility
This keeps one owner for runtime semantics:
  • PersistentRuntimeDriver::recover() owns input/runtime/control recovery
  • MeerkatMachine owns session entry recovery (ops_lifecycle, epoch_id, cursor state)
Direct substrate usage is intentionally narrower:
  • Queue-only turns
  • no runtime-owned keep_alive
  • no Steer/render-metadata semantics
  • no runtime ingress/admission ownership

Agent construction contract

Agent construction is centralized in AgentFactory::build_agent().
  • Surfaces pass per-request build data in-band via CreateSessionRequest.build (SessionBuildOptions).
  • No out-of-band staging lock is used.
  • FactoryAgentBuilder maps SessionBuildOptions to AgentBuildConfig.
  • For runtime-backed builds, SessionBuildOptions.runtime_build_mode should carry RuntimeBuildMode::SessionOwned(bindings) from prepare_bindings().
  • For standalone/testing/embedded builds, prefer RuntimeBuildMode::StandaloneEphemeral explicitly.
  • Session metadata persists realm context and durable session identity:
    • realm_id
    • instance_id
    • backend
    • config_generation
    • durable LLM identity
    • keep_alive
    • visible comms identity metadata such as comms_name and peer_meta

Session lifecycle and turn semantics

create_session

create_session(req) builds the agent and optionally runs the first turn.
  • Returns RunResult with session_id.
  • InitialTurnPolicy::RunImmediately executes the first turn inline (default).
  • InitialTurnPolicy::Defer registers the session without running a turn.
Create has a commit boundary:
  • pre-commit failure: side-effect free; no committed session identity
  • post-commit first-turn failure: return session identity and keep the session resumable
Committed create failure must not be silently rewritten to “cancelled” or cleaned up as unpublished work.

start_turn

start_turn(id, req) executes a new turn on an existing session.
  • At most one in-flight turn per session.
  • Concurrent attempts return SESSION_BUSY.
  • A turn the runtime queue does not dispatch (a direct turn, such as a create’s first turn) also returns SESSION_BUSY while the session’s runtime holds a run that waits to resume: one a store fence gap suspended (see Durable runs), or one a restart left open. It goes through once the runtime has resumed or settled that run.
  • Committed success must not be rewritten to cancellation.
  • Runtime-backed surfaces may hot-swap supported live settings on an existing session where the surface contract says that is allowed.
  • Runtime-backed surfaces do not race start_turn: they admit each turn through the session’s runtime queue, so a surface turn that arrives while another is running waits behind it and then runs. From 0.9.15 one kind of JSON-RPC turn is the exception: a turn that would change the session’s model settings beside a provider-neutral reasoning effort level is answered SESSION_BUSY at once while the session has unfinished work (see Reasoning effort).
The runtime may coalesce multiple queued Steer inputs into one agent turn. Admission remains durable per input, but there is no promise of one model call per input. Applications that require one side effect per admitted input must iterate the durable input records deterministically in a tool or host-owned worker. Do not rely on the model to notice and act once for every item in a batched prompt. Inputs of different original requesters never share a turn: the runtime coalesces only inputs that carry the same requester, or none, and a steer from another requester waits for a turn of its own instead of joining a running one. Steers keep the order they were admitted in across requesters: a steer waiting for a turn of its own keeps its place, so a later steer of the running turn’s own requester waits behind it too rather than join the running turn.

Ordered System messages

system_prompt on StartTurnRequest appends one ordinary, unkeyed System message at that turn boundary. System messages are repeatable and may appear anywhere in the transcript; an append never replaces row zero or rewrites an earlier message. A resume that names an explicit system prompt appends it as such a row too, except, since 0.9.9, when the prompt is exactly the newest System row the model reads as a prompt (one without an append identity, an instruction activation or an operator-instruction entry, that is not a superseded prompt version): then it appends nothing, where earlier releases appended another identical row. A different prompt, and an earlier prompt sent again after a change, still append; a bare resume appends nothing; and StartTurnRequest.system_prompt keeps its per-turn append. A host that materializes a resumed session itself asks Session::newest_prompt_row_is. Keyed system-prompt updates instead append rows carrying SystemPromptVersionIdentity: a SystemPromptKey and SystemPromptVersion. The active model view selects the latest version per key; unkeyed rows are never superseded. Older versions remain historical rows reachable through retained transcript revisions, not a promise to retain every version in the active model view. There is no host-configuration injection ledger. Materialization and resume never infer a System message from current host configuration. A host that wants to add an instruction supplies it through the admitted update path, where it becomes a new ordered System row rather than an overwrite. A host that wants earlier ordinary System rows gone (prompts that earlier boots wrote, for example) names them to the system-row prune, SessionServiceTranscriptEditExt::prune_system_rows: one audited rewrite, reason kind system-row-prune, removes exactly the named rows. Each row is named as a history read showed it, by its 0-based message index and sha256:<64 lowercase hex> over its content as UTF-8 text; Core never picks rows, and a stale reference is refused typed with nothing written. Only ordinary System rows are removable: a keyed prompt version, an instruction activation row and an operator-instruction entry row each keep their own lifecycle, and a row an operator-instruction entry follows in its group of System rows stays. A session running a turn answers Busy; a dormant one is not woken, and no model is called. The prune can also retire superseded instruction activation bodies in the same commit. Compaction retains active keyed and unkeyed System rows verbatim in relative order, while discarding superseded keyed versions from the compacted active head. This does not rewrite their retained revision history. Instruction activation rows keep their places too; the one change a compaction makes to one is to retire the body of an activation row that a later activation of its key superseded, which then reads as a header that names its key, revision and body length (see Durable Instruction Activation). Provider adapters lower the active instruction view without rewriting durable history. Wires accepting ordered role=system messages normally preserve their positions. Anthropic’s model-gated lowering can instead place turn-scoped instructions after their governing User and before the Assistant, subject to its legal predecessor constraints; this is a wire-only placement adjustment (see System-message projection). Providers with only a top-level instruction field reject later System rows with a typed projection error; unsupported placements on other wires also fail typed rather than silently hoisting or merging canonical rows. Provider capability never changes, rejects, or rewrites the durable Session itself.

Operator instructions

Since 0.9.9 a host gives a session standing System-plane instructions by slot (a namespace and a key), revision and body. Since 0.9.11 they are on JSON-RPC (session/set_instruction, session/revoke_instruction and session/instruction_status; see RPC operator instructions) and in the Python and TypeScript SDKs; in Rust, SessionServiceInstructionExt::set_session_instruction records a session-owned write (activate a body, or revoke, at a higher revision; an exact retry answers as a duplicate), and read_session_instructions reads the recorded target, the entries in effect and how the session’s model lowers them (lowering). They replace instruction activations (session/activate_instruction), deprecated since 0.9.11. The persistent service records a live session’s write through a task it owns and a dormant session’s store-only, without waking it; the ephemeral service refuses durability_unavailable. A write changes only the recorded target. The pending changes apply at the session’s next fresh run start as one System entry appended before the run’s own input, naming the run; a resumed run applies nothing, and a run that never starts takes its entry back, so the changes stay pending. A model that lowers System rows only in its leading prompt defers them (leading_prefix_only) until a compaction at a run’s first request carries them in its snapshot, and a backend whose wire carries only one leading System row keeps them pending (lowering_unavailable). A compaction folds every entry row into one snapshot of the instructions in effect after the leading System prefix, and no summary request, host curator window or compactor input carries an instruction body. Each applied entry is published as an operator_instructions_applied event: a delta once the run whose start appended it starts, a snapshot once its compaction commits (a staged memory projection’s when the runtime commits it, right after its compaction_completed; a compaction rolled back before its commit is never observed). The event reports what is now in effect for the live session, not durability: a restart before the entry is durable applies the changes again and publishes the event again. The status read is the authority. A stale or conflicting revision, a slot that instruction activations manage, more than 64 slots, a body over 256 KiB and more than 1 MiB of Active bodies are refused typed (InstructionSetErrorCode), and so is a model change onto a target that cannot carry the entry rows the session wrote where they stand. On a governed runtime (a runtime with a native work authorization host) every operator-instruction write, session-owned or a mob’s, is refused prepared_authorization_required and nothing is written: such a write needs prepared authorization, which this release does not mint, so a retry is refused the same way. A mob member’s session belongs to its mob, which writes through MobHandle::set_operator_instruction (mob/set_instruction): the session’s own host’s writes are refused owned_by_mob (see Operator instructions). A session that holds operator-instruction facts is written at session envelope 5, which releases before 0.9.9 refuse (see Upgrading).

interrupt

interrupt(id) cancels an in-flight turn.
  • If no turn is running: SESSION_NOT_RUNNING. A runtime-backed session whose run a store fence gap suspended (or a restart marked resumable and has not resumed yet) has no turn running, but its interrupt is accepted: it records the run’s stop, the run never resumes, and its inputs end cancelled once the session’s store takes writes again.
  • A runtime-backed session whose store’s writer is stopped (a store fence gap) has its run suspended by the interrupt, not hard-cancelled: the stop is recorded the same way, since the gap would refuse the cancel’s writes.
When a service drops an active agent execution future, cancellation reports both the cancellation request and the observed end of that execution to the generated turn authority. The resulting terminal effect lets a standalone session accept a later turn; merely entering Cancelling is not terminal completion. For surfaces using SurfaceRequestExecutor, upgrading a cancellation action during initialization is serialized with request cancellation. Cancellation either receives the new action or the installer replays it for an already cancelled request. Actions run outside the owner locks and may re-enter the executor; this does not promise rollback of effects already executed.

read and list

  • read(id) and list(query) are non-blocking with respect to in-flight turns.
  • Persistent services can include durable sessions from the realm backend.
  • Presence in list() is not the same as “live session exists”; runtime-owned seams must answer liveness.

read_history

read_history(id, query) returns the last committed transcript snapshot for a session.
  • Messages are returned oldest-to-newest.
  • offset and limit apply from the start of the full transcript.
  • Active sessions do not expose in-flight partial output through history reads.
  • Archived sessions remain readable on persistent backends; the ephemeral service rejects archived history reads with SESSION_PERSISTENCE_DISABLED.

archive

  • The canonical SessionDocumentMachine owns the session_lifecycle_terminal fact for ALL profiles; shells realize its verdict, they never decide it.
  • Realization order is fail-closed: durable document commit first, runtime retire second. RuntimeState::Retired is the runtime realization of the same verdict, so the resurrection window is unrepresentable.
  • If runtime retirement fails after the document commits Archived, the durable terminal remains authoritative: reads, resumes, and turns stay rejected while a repeated archive command follows the generated convergence transition and retires the residual runtime.
  • Archived sessions are excluded from list() and rejected for further reads/turns (SESSION_NOT_FOUND on persistent services; the ephemeral service serves a final in-memory view).
  • Committed transcript history remains readable via read_history on persistent backends.

Keep-alive contract

keep_alive is a runtime/session concept, not an old “host mode” execution path. Rules:
  • create / run omitted => default false
  • continue / resume omitted => inherit persisted session intent
  • explicit keep_alive override is a session/runtime mutation once validated
  • invalid keep_alive + comms configuration is rejected before any stateful work
  • once validated, an explicit keep_alive mutation may commit independently of turn success
keep_alive=true requires usable comms identity/config on the surfaces that expose it.

External events

External events are queue-only runtime-backed inputs.
  • they are admitted into runtime ingress
  • they do not invent a second direct execution path
  • “turn-boundary inbox draining” is not the primary mental model for the current runtime-backed design

Event delivery and durable projection

Live delivery and durable evidence use separate lanes:
  • Ordinary session subscribers read a 256-entry broadcast. If a subscriber falls behind, it receives a synthetic StreamTruncated event with reason.kind = "stream_lagged" and the exact reason.dropped count, then continues with retained events. The marker reports a gap in that subscriber’s live view; it is not a canonical durable-log event.
  • A persistent session whose host installs event projection has one internal dedicated projector stream backed by an unbounded MPSC queue. Built-in realm-backed persistence installs it; custom hosts must call the projection composition seam explicitly. Event publication shares the same envelope with the live lane, but projector latency does not block live subscribers or depend on broadcast retention. The runtime emits a rate-limited degraded warning when the queue reaches 1,024 entries and records its high-water mark.
  • The projector asynchronously appends each envelope to the durable event audit log before updating derived .rkat/ materialized files. UI-ring lag does not drop projector input, but EventStore projection is still best-effort derived state. The RuntimeStore/backend carrier remains session authority; SessionStore rows and EventStore are component/content and projection seams.
A durable audit-log append failure latches an EventProjectionHalted fault, records a durable projection-halt marker when possible, stops before appending later events across the sequence hole, and makes replay and later resume fail closed. It does not roll back or fail the already-committed turn. Clients that see live stream_truncated reconcile from events/list_since, a current snapshot, or ATIF export. Every committed close of one of the session’s live channels publishes live_channel_closed on the session event stream, after the close commits: session_id, channel_id, a typed reason (LiveChannelCloseReason: client_requested, client_disconnected, provider_closed, error, media_fault, replaced or open_abandoned) and reopen_recommended, true only for a media_fault while the session’s one reopen is still available. Observers learn the cause from the stream instead of polling live/status.

Assistant message identity

Every assistant message the agent loop commits carries a session-scoped assistant_message_id (0.8.45+). The same value is on the committed block_assistant history row and on every live event for that message, so a client pairs live rows with canonical history by id alone. It never has to compare text or rank, and identical answers stay distinct. Assignment. The id is minted in the provider-neutral agent loop when a provider turn starts, before hooks run, before the request is sent and before any delta. It is opaque: never derived from content, and not a timestamp (native builds mint UUIDv7, browser builds UUIDv4). Assignment happens for all providers; the id never enters the provider request. Events that carry it. turn_started, reasoning_delta, reasoning_complete, text_delta, text_complete, server_tool_content, assistant_image_appended, turn_completed and retrying name the message they stream or close. run_completed.assistant_message_id names the committed message whose text result repeats. That may be a message from an earlier run (for example when a run ends on budget exhaustion before committing new text). Rust consumers can read any of these through AgentEvent::assistant_message_id(). Commit and failure.
  • Commit is a history fact: a message is committed exactly when a block_assistant history row carries its id.
  • turn_completed with the id is only the live signal that the provider turn finished. It proves neither commit nor durability, and its absence proves nothing either:
    • It can come BEFORE work that can still fail the run: the boundary work after a tool-use turn (turn-boundary hooks, the comms drain), the run-completed hooks of a run with an output schema, and the session save. A live display-text drain on a persistent session publishes its turn_completed before the session is persisted; if persisting fails, the id never reaches history.
    • It can be ABSENT for a committed row: a run that fails after the row was pushed but before turn_completed was published (for example a terminal-turn boundary hook or a run-completed hook denies) keeps the row and its id, and no turn_completed follows.
  • So when a run ends (run_completed, run_failed, a truncated stream or a reconnect), read history for every id you saw: a row with that id means the message was committed, no such row means it was discarded. History read after the run ends is the only authority.
  • An id is on at most one committed row. A failed provider turn (the request fails, retries are exhausted, max_tokens is hit while thinking, or a pre-request hook denies the call) never appends a row, so its id is never committed.
  • Retries of the same provider turn reuse its id: same-model, empty-output, stall and timeout retries, and model fallback. retrying carries the id; discard the live buffer held for it and render the retry’s deltas in its place.
  • A re-poll after compaction re-emits turn_started with the same id. A repeated turn_started for an id that is still open restarts that message’s live buffer.
Which rows get one.
  • Each provider turn’s assistant message, including tool-use turns and structured-output extraction turns. Extraction turns emit no turn_started, text_complete or turn_completed; their deltas and row still carry the id.
  • An assistant message appended by a tool effect gets its own fresh id, with no turn_started. Its images are published as assistant_image_appended under that id. The id of a message appended after a callback tool result is fixed when the batch is applied, so replays and redeliveries return the same id.
  • A row committed by a live channel’s display-text drain (host-produced assistant output appended through append_external_assistant_output) gets a fresh id that core mints when it commits the row. The text_complete and turn_completed published for that commit carry it. There is no turn_started for it, and its streamed text arrives as realtime transcript deltas, which carry no assistant_message_id. A drain that commits no row (no display text) publishes turn_completed without an id.
  • No id: compaction summaries (user rows), rows the realtime transcript materializer commits for a provider-managed realtime turn, and any row written before 0.8.45. Old rows and events are never backfilled.
  • Realtime rows pair by provider item id instead. The text_complete and turn_completed published when the materializer commits a realtime turn carry no assistant_message_id, so a turn_completed without an id does NOT mean no row was committed. The history row carries realtime_origin (on the wire since 0.8.45, the same RealtimeMessageOrigin shape run events carry in identity), and the live transport’s realtime observations carry the same provider item ids: provider_item_id on assistant_text_delta, assistant_transcript_delta, assistant_transcript_final and assistant_transcript_truncated, and item_id on assistant_audio_chunk and inside realtime_transcript events. A console that rendered live text for a provider item replaces it with the row whose realtime_origin.provider_item_ids contains that item id. One row can cover several items (a barge-in that re-keys speech into a second item). The id-less text_complete and turn_completed carry no provider item id either, so pair realtime rows with the live transport’s observations, not with session events. The origin is set when the live host hands the session an assistant event together with the channel it streamed on, which the built-in RPC and facade projection sinks do. A realtime row committed by an earlier release, or from events a custom LiveProjectionSink applied without LiveTranscriptIdentity::channel_id, has no realtime_origin; render it from history only. Core does not mint an assistant_message_id for these rows: their live output reaches the client straight from the live transport, never through the agent loop, so no core-minted id could be stamped on it before the row exists, and the provider item id already is the stable key both sides share.
  • Deltas with no id are output that is not a transcript assistant message, such as the compaction summary call. They also come from a legacy custom client, or a client decorator that does not forward AgentLlmClient::prepare_request_attempt. A decorator MUST forward prepare_request_attempt to the client it wraps (passing its transformed messages); otherwise the loop falls back to the legacy streaming path and the live deltas behind it carry no id. The loop still stamps turn_started, text_complete, turn_completed, retrying and assistant_image_appended in those cases.
History, compaction and rewrite.
  • Paging: the id is on each row, so any offset/limit page carries it. It implies nothing about position, and history can be read before the final live chunks of the same message arrive.
  • Digests: the id is construction bookkeeping. Transcript revisions, rewrite span digests, continuity guards and the provider-cache prefix identity are identical with and without it.
  • Compaction: retained rows keep their ids byte-for-byte (core validation already requires retained rows to be unchanged). Discarded ids leave the active transcript, are never minted again or reused, and stay readable through read_transcript_revision(parent). The summary row has no id. Reloading a session (whole-blob or head-canonical storage, crash-recovery tail) round-trips rows unchanged.
  • Generic rewrite: replacement rows never carry an id. Core clears any supplied value, and wire rewrite input does not accept one (an echoed history key is ignored). Rows outside the selection keep their ids. A replaced or removed id is gone from the active transcript but stays readable in the parent revision.
  • Restore: session/restore_transcript_revision re-installs rows core reads from the session’s own retained revision, not caller-authored rows, so every restored assistant row keeps its id verbatim. Restoring the revision a rewrite or compaction replaced brings its ids back into the active transcript.
  • Forks: fork, fork_at and fork_at_complete_boundary keep row ids (they are the same occurrences; the child session never re-emits them). fork_replacing keeps the inherited prefix’s ids and clears the id of the edited row, whether it replaced a whole message or one assistant block.

Explicit override semantics

Where the surface supports omission vs explicit override, these are distinct facts:
  • omit / inherit
  • disable / false
  • set / concrete value
When all three meanings matter, the wire/API must preserve that distinction. Typed optional fields and override masks are preferred over default-value folklore. Concretely: SystemPromptOverride (Inherit / Set / Disable) is the canonical wire+persisted carrier for the per-request system-prompt fact, and per-turn provider parameters travel as the typed ProviderParamsOverride, not a raw JSON bag.

Realm contract

Sessions are realm-scoped.
  • Same realm_id plus the same physical provider/root: shared visibility and config context.
  • Different realm_id: strict isolation.
  • Backend is pinned per realm via realm_manifest.json (manifest format 2 adds provider and ephemeral_domains; readers refuse formats newer than they support instead of silently ignoring fields, and realms pinned to an external storage provider refuse built-in disk opens typed).
  • State roots resolve realm-id-first across the project-local and user-global candidates; a realm materialized under both is a typed split-brain refusal (see rkat storage doctor).
  • First materialization of a fresh realm reserves across the candidate set resolved by that invocation before writing the manifest. Processes using the same multi-root set converge; surfaces probing different sets do not. Use one explicit root or matching server context for cross-surface starts.
  • A fresh prompt-first CLI run or rkat help whose workspace-derived realm cannot open may isolate storage into a newly generated durable SQLite realm under the same resolved state root. No historical session from the failed realm is loaded into the fresh run, and the compatibility bridge is never invoked automatically. An ordinary supported initialization or migration may have completed before a later store refused. The original workspace config and auth/tool policy still govern the run. Explicit realms, isolated runs, resumes, and session commands never take this fallback. Historical session access requires the explicit fenced pre-0.8.10 bridge; there is no in-memory compatibility fallback.

Mob-member role continuity

A durable mob member normally resumes only when its stored and requested role match. Mob ID, member AgentIdentity, bridge session_id, and transcript are fixed identity facts; a profile edit alone cannot reinterpret the stored member as a different role. Trusted in-process hosts have one explicit migration seam. An exact MemberLaunchMode::Resume may carry resume_from_role naming the stored predecessor role for that request. Admission proves that the stored and current comms_name and MobMemberBinding refer to the same mob and member. An absent declaration returns MemberRoleMigrationRequired; the wrong predecessor or a contradictory identity returns MemberRoleMigrationRejected. Role migration is cold and one-shot. The exact session must have no live actor, the declaration is consumed rather than persisted as standing permission, and successful materialization restamps the current role into durable comms metadata while preserving the same session and transcript. There is no implicit rollback; a later role change requires a new exact cold migration. This seam is deliberately private. Public CLI, REST, JSON-RPC, MCP, generated SDK spawn requests, helpers, and standing profiles cannot set resume_from_role. The private member-host materialization protocol carries it only as part of the trusted remote-host resume path.

Member creation provenance and host activation

The canonical MemberSpawnedEvent.creation record carries a runtime-issued MemberCreationId and typed provenance. A fresh, unrelated member receives a new token even if it reuses a retired member’s display identity. Cold restore preserves the token. An explicit runtime-authorized successor preserves the original token and birth cursor and records its exact predecessor binding. SpawnMemberSpec::new defaults to Unproven. Trusted host ingress must use SpawnMemberSpec::host_root explicitly. Agent-lane handles downgrade that claim; authenticated source witnesses establish delegated ancestry instead. Policy auto-spawn remains Unproven, because a host-configured policy does not prove that the resulting creation was independent of member activity. RPC and REST spawn ingress also remains Unproven until it carries an authenticated originating principal. Transport choice alone is insufficient; that authority work is tracked in #1646. Historical creation facts remain in the existing journal-backed roster projection after retirement. MobHandle::member_creation_for_session reads the facts for an exact session, and member_creation_journal_cursor exposes the committed journal position. These reads do not authenticate a current caller. Hosts must independently validate its current session/member binding. Conflicting history is an error, not evidence that an ancestor is absent. MobHandle::capture_member_creation_source issues a sealed process-local MemberCreationSourceWitness after checking the current and persisted typed binding. Agent arguments, transcript text and comms labels cannot construct that witness. Unsupported source authority remains unknown. A delegation edge records ancestry only; it never grants execution rights or resource access and carries no serialized tool policy. Current tool execution policy and host access policy still apply to every call. Optional consumers that need a source policy ceiling must read it through the native session API; missing proof cannot be interpreted as unrestricted access. MobSessionService::load_retained_session_metadata is the supported exact-session read for ancestry and policy-ceiling observations. Its PersistedSessionMetadataView contains the latest committed metadata of the exact retained session, including archived sessions. Persistent services read through the authoritative native metadata owner. An absent session returns None; corrupt metadata returns an error. Unsupported backends return a typed Unsupported error. The method never substitutes a successor, revives a session, permits a write, or changes ordinary load_persisted_session_metadata visibility. It does not provide a historical revision, lease or currentness guarantee. A valid present metadata view whose resolved tool policy is None has the native meaning of unrestricted; missing metadata or an unresolved policy must not be interpreted that way. An optional resource service can intersect its current ACL with the current policies of the exact source and executing sessions. In this dynamic model, an authorized host policy change can widen an existing resource-access route; the native policy owner records that change. An ancestry record is never a frozen policy ceiling. Native tool execution still makes its own decision. These observations identify an executing agent, not the original human requester, and are not an execution-time currentness fence. Native originating authority and governed execution work continue separately in #1646 and #1730. Trusted hosts can install MobBuilder::before_activation to bind optional services before fresh or restored members execute. The MobBeforeActivation callback receives a MobReadHandle backed by the same roster and lifecycle watch as the eventual public handle. Only direct read operations are exposed; the type cannot submit actor commands while the actor has not started. A callback error aborts bootstrap. Hosts must discard provisional service bindings if a later bootstrap step fails. With no callback configured, no host hook runs. MobMcpState::set_before_activation carries that hook into delegated child mobs. set_additional_child_tool_bundles carries the matching optional tool factories, so children can use a configured host service. Each factory uses the actual executing caller’s resolver; registration conveys no inherited parent authority and exposes no tools unless the host selects their bundle. A bundle registered ChildOffered reaches only a child profile that names it in tools.bundles, and only while the host offers it (see the mobs guide). MobMcpState::set_after_destroy is the release counterpart for the mobs that state destroys. Once a destroy completed natively and the mob left the managed set, its MobAfterDestroy hook gets the mob’s id before the storage cleanup the destroy owes, and again before each retry of an outstanding cleanup. A host drops the read handle it kept there: a handle still held keeps that cleanup outstanding, and the destroy answers Incomplete. Another mob may be managed under the same id by then, so the host releases only what observes the destroyed mob. The hook must not block; a panic in it is caught and logged, and the destroy goes on. An observer drain that other owner handles refuse is retried for about a second, the hook running again before each try, so a holder that lets go shortly after the hook does not leave the cleanup outstanding. MobMcpState::set_before_create is the admission counterpart for the mobs that state creates. Before anything of a new mob exists, its MobBeforeCreate hook gets a MobCreateRequest that the create path builds itself: the mob id, the owner bridge session (absent only for a host’s own create) and whether the mob is that session’s implicit delegation mob. From 0.9.15 the request also carries council, which is set only for the mob a temporary council creates for itself. council names the council id and, in convener_session_id, the session whose council tool call convened it: the session that tool’s surface is bound to, as the council recorded it at its first admission. A council a host runs directly through the Rust API has no convener. A council’s owner bridge session is a session of the council’s own, so the convener is the one field that says who asked for the mob. No tool argument and no request on any surface sets council or its convener. Every create path asks it; restoring a persisted mob creates nothing and does not, and neither does a handle a host registers itself with mob_insert_handle. The hook fails closed: a refusal or a panic refuses the create with an error MobMcpState::is_create_refusal recognizes, and the reason is in its message and structured data.

Compaction contract

Compaction is optional. Ordinary recoverable failures preserve uncompacted history and normally report CompactionFailed, allowing the run to continue.
  • Triggered by token thresholds and turn guards, by the projected input of the next request, and once after a request that did not fit: one the provider refused for not fitting the model’s context, or one that carried the open run’s own tool results shortened.
  • Also triggered once by a host’s compaction request (MeerkatMachine::request_compaction, JSON-RPC session/compact and mob/compact_member): the session’s next check consumes it and compacts whatever the thresholds and cadence say. A check that consumes it and compacts nothing (before the session’s first model request, or with nothing to fold beside the open run’s request and newest round) publishes requested_compaction_skipped with the reason.
  • A request still over the provider’s request-body cap or the model’s input budget after the compaction check carries the open run’s own oversized tool results shortened for that request only, and publishes tool_results_bounded_for_request; the stored transcript keeps them whole.
  • A typed provider PolicyStop from the summary call terminates the run.
  • Unresolved durable compaction-stage authority or cleanup errors, such as SessionDurableProjectionAuthorityUnknown, also terminate fail-closed.
  • Preserving history does not guarantee the next request fits the provider’s capacity; an unrepaired oversized request can still fail.
  • Compaction usage counts toward run budgets.
The staged transcript-rewrite/memory transaction is distinct from the optional asynchronous event-audit projection. An EventStore audit append failure does not undo an already-committed turn; that guarantee does not permit ignoring uncertain compaction-stage authority.

Durability

Ephemeral mode

No durability across process restart.

Persistent mode

Durability follows the selected realm storage provider and its declarations for the seven owned domains. The built-in persistent realm backends are sqlite and jsonl; an external RealmStorageProvider may supply other persistent stores, but it must declare every domain and may not silently substitute memory for one declared durable. Both built-in persistent backends mount a SQLite runtime companion for runtime authority: sqlite realms keep the runtime_* tables inside sessions.sqlite3, and jsonl realms carry a dedicated runtime.sqlite3 alongside the JSONL session documents. Queued inputs, run-boundary receipts, ops snapshots, and auth-lease/OAuth-flow authority are durable under both backends. The memory backend is explicit ephemeral storage and does not survive process restart. Since 0.9.5 a host can also keep a realm’s durable state in its own database through the host store SPI.
  • Completed turns are persisted.
  • Since 0.9.5 a run commits after every tool round, and after a crash the run continues from its last committed round under the same run id (see Durable runs). A crash loses at most the round in progress; any durable tail is preserved and recovered (see below), never discarded.
  • SQLite-backed realms are the default persistent mode and support normal same-realm multi-process workflows. Declare [storage] hosting = "multiprocess" so each session is hosted by one process and deliveries route to it (see Realms: multi-process hosting).
Persistent session metadata is the source of truth for resumed durable session behavior unless an interactive surface supplies an explicit typed override. A per-build compaction threshold (SessionBuildOptions::auto_compact_threshold_override) is not part of it: it holds for the build that names it, so a recovered or resumed session is built without it unless its host supplies it again, while realm [compaction] config and a mob profile’s auto_compact_threshold apply to every build.

Domain document and store authority

The current Session envelope is version 4 and contains domain state only. It does not authenticate itself and does not carry a persistence token. A deserialized Session cannot authorize a write, migration, or recovery operation. The selected store profile issues the physical authority:
  • WholeBlob - session_id, monotonic store revision, and SHA-256 of the exact serialized row.
  • HeadCanonical - session_id, monotonic store revision, the committed boundary head, and its exact head token. The boundary head binds the message-row, rewrite, graph, component, and metadata prefixes.
Version 3 remains a supported read format. The generated persistence version authority restores it to version 4 in memory; new Session and physical-head writes use version 4. HeadCanonical reads preserve the original persisted v3 head and CAS token. Retained runtime boundaries and provisional references keep that exact identity until normal writes advance them. Reading alone does not rewrite the envelope. A reader that accepts only v3 refuses new v4 data. This is the compatibility boundary that preserves deferred callback infrastructure failures across continuation. After a v4 write, rolling back to a v3-only reader is unsupported; do not change the version field or remove failure metadata to make it load. RuntimeStore also maintains a small catalog entry in the same atomic boundary as physical authority. The catalog can answer listing and lifecycle discovery, but cannot carry a transcript, graph, component body, or serialized Session. It is not a second session document. Meerkat 0.8.11 accepts durable state from 0.8.10 only through the one-time released-state importer. An exact released v2 document with the historical embedded proof is checked once by the frozen 0.8.10 verifier. An unstamped released document is accepted only when the same store transaction proves the released physical schema and exact source row or blob identity. Either path strips the historical proof fields and installs current store-issued authority; ordinary v3 reads never enter the importer, and no current write emits the old fields. State older than 0.8.10 in a SQLite realm is admitted only through the explicit offline command rkat ... storage migrate --apply --bridge-pre-0-8-10. JSONL and memory realms are rejected before any database is mutated. This is a frozen, fail-closed pre-floor importer: under the realm maintenance fence it authenticates supported historical schema before installing current representation and authority. Ordinary store opens and v3 reads remain strict and never enter the bridge. Catalog authentication is per domain and row admission is per record: a row the current typed contract cannot represent without loss is left byte-identical on disk and reported, rather than costing the realm every other row. The maintenance transaction preserves input rows - including their ingress payloads, whatever lifecycle state they reached - without scheduling or replaying them. A later session activation follows normal recovery, which applies this binary’s ordinary terminal-payload retirement to rows it writes itself. A SQLite realm activates the HeadCanonical profile when it opens: SqliteRuntimeStore::new_head_canonical converts each WholeBlob runtime from the co-located session store’s own row for its session, and refuses typed when that store holds no row, because in a realm a missing row can be a deleted session that activation must not bring back. A standalone WholeBlob runtime database - one a host composed with SqliteRuntimeStore::new on a file of its own while its session store lived elsewhere, such as a Meta 0.9.3 persistent gateway’s runtime.sqlite - holds no physical session, so it converts once, offline, through SqliteRuntimeStore::convert_standalone_whole_blob_in_place. The host first opens SqliteSessionStore on the same file: without the co-located session schema the conversion refuses typed and leaves the database file unchanged. Before it writes, the conversion checks every runtime it would convert, in the order it converts them: a WholeBlob body that is missing, differs from its digest, or does not lay out as the session store’s rows, or a session the guard below refuses, refuses typed before anything is written, so a file the conversion has not pinned stays a WholeBlob database. The conversion then pins the file and seeds each runtime’s session that the co-located store holds no row for from its digest-verified WholeBlob body, in the transaction that installs the runtime’s authority and completes its activation receipt, so the co-located transcript, rewrite history and metadata are that body’s. A body whose transcript graph retired its oldest occurrences seeds the rows the store keeps after retiring the same occurrences: the retired base and rewrite rows for the retained occurrences only. It seeds only while the co-located store holds no session beyond the ones the file’s completed activations own: any other session refuses that runtime typed without seeding, and runtimes to seed convert before runtimes whose session already has a row, so a session the store held before the conversion always refuses. The host therefore writes no session row into the file before the conversion returns. A conversion stopped part way finishes on the next call, and a converted file opens with new_head_canonical. The caller asserts that the file is standalone: the conversion is never used on a realm database, and the guard cannot tell a realm whose remaining sessions all completed activation from a standalone file.

Durable-tail recovery

The intra-turn persistence hook writes a provisional physical successor outside the runtime boundary transaction. Every successful write returns an explicit RunCheckpointReceipt. Receipts are scoped to one session, committed base, and run; candidate_sequence starts at one and advances contiguously. An exact retry may return the same receipt, while the next candidate must present and succeed the preceding receipt. A stale, divergent, or out-of-sequence candidate fails closed. WholeBlob encodes, hashes, and writes each candidate once. HeadCanonical applies one sealed prepared delta and returns the exact resulting physical revision and head token. The actor retains only the latest successful receipt; an uncertain outcome or failed acknowledgement terminates the actor and requires durable reload. At the final boundary the store promotes that exact candidate. It does not construct a second document or replay the delta. Since 0.9.5 every tool round’s second commit is an intra-run checkpoint (see Durable runs); the round’s first commit stages the model’s response in the session’s run cursor and writes no transcript row. On HeadCanonical each such checkpoint moves the durable head over the run’s rewrites, so a run that compacts more often than history_retained_rewrites keeps retires its live history to the retention. Such a checkpoint’s tail always ends with the run’s own tool call and its results, never with a compaction summary. Intra-run checkpoints never prune stored rewrite rows, so the committed boundary stays rebuildable for recovery; the boundary commit that promotes the run applies the run’s retention cut in the same transaction, never past the promoted head’s row-lineage anchor. On WholeBlob the run’s first round commit after a compaction re-bases the run: one store call commits the session document as it stands at that commit, the compaction included, as the session’s committed base and restarts the run’s provisional tail on top of it, so round commits continue after any number of compactions. A crash between the compaction and that commit continues the run from its previous commit, and the compaction runs again. A later failure of a re-based run consumes its inputs instead of requeueing them, because they are already part of committed history. A crash after an intra-run checkpoint continues the run (see below); before 0.9.5 it closed the run as interrupted. A crash or shutdown race can still leave a durable transcript tail (up to and including a fully completed turn) whose runtime boundary commit never landed. Durable is not the same as runtime-committed, and neither view is silently preferred: a cold read drives a typed, machine-owned read-source decision (serve the committed runtime snapshot, serve the committed store head, recovery required, or quarantine) instead of a shell comparison. The recovery rule is never-discard: every store-proven durable descendant of the committed authority is preserved.
  • Since 0.9.5 a continuable tail is not closed. A tail is continuable when it carries the run cursor of the open run and agrees with it (after a round’s first commit, the staged response is not in the transcript yet), every input bound to the run is still bound to it, and the runtime is not retired. A tail with no assistant row, such as one written by the first commit of a run’s first round, by the first commit of the round after a compaction, or by a WholeBlob re-base, is continuable only on that cursor’s evidence: it holds no tool result, the cursor names the store-issued run id with none of its staged calls in the tail, and the store proves the tail’s descent. Recovery then writes nothing and settles nothing, and the run continues from its last committed round under the same run id (see Durable runs). A cursor never turns a tail that would otherwise be held into a continuable one, and a cursor inside a committed body never makes a run continuable. A cursor tail whose bound inputs were settled, whose run has no bound inputs (a direct or machine-service turn), or whose runtime was retired is closed as interrupted. So is the tail of a run settled without continuing: stopped before it continued, refused by a durable worker’s resume gate, or kept by a requeue stop past its max age. The abandonment of its inputs commits first, then the tail is closed as interrupted, never discarded. A run that fails and whose inputs are queued for another attempt discards its tail before they requeue, so it starts over from its inputs; a cursor tail left behind with requeued inputs is closed as interrupted, and the inputs its cursor binds are consumed with it, so they do not run twice.
  • A completed tail (single run, EndTurn terminal, structurally coherent) is committed as a recovered run boundary in one atomic store transaction: recovered snapshot, boundary receipt, and input terminalization land together, so exactly-once holds and the delivery layer never re-runs a recovered turn.
  • An interrupted tail is closed as interrupted: content preserved, a typed recovery notice appended, the original run terminalized, never requeued. Since 0.9.5, when the tail’s run cursor holds a round’s staged model response, the response is written into the transcript before the notice, with an outcome_uncertain tool error result for each of its tool calls, which says only that the call’s outcome is unknown: the calls may have run.
  • Since 0.9.5 a tail whose run cursor agrees with it settles the inputs its cursor binds: the cursor binds exactly the inputs whose content the run wrote into its turn, and a committed receipt of the run names more. Those not ended yet are consumed with the tail, whatever run they name now. A content input the cursor does not bind is not in the tail: one that names no run, or the tail’s run (a steer that joined after the last round commit), stays queued and runs once after the tail is closed; one that names another run holds the tail for reconciliation. A tail without an agreeing run cursor (none, or one the classifier finds stale or cannot check) settles the inputs that name its run and holds beside any other content input. A cancelled run (an interrupt or a run stop) is closed as interrupted when its cancellation commits, under these rules.
  • Anything else, including any tail carrying a dangling tool_use, is held intact for reconciliation. A dangling call proves intent, not execution; its external side effect may already have fired, so recovery never closes it with synthetic results. A tail whose last turn stopped for a reason other than EndTurn or ToolUse (for example MaxTokens or PauseTurn) is held the same way: that turn did not reach a natural end. Since 0.9.5 an operator can resolve a tail held as ambiguous: rkat session resolve-held-tail commits it as interrupted, keeps every row, and answers each call without a result with an outcome_uncertain tool error, which says only that its outcome is unknown (see Storage Operations).
A session whose transcript has rewrite occurrences (compaction) is recovered the same way: a completed tail becomes a recovered run boundary, an interrupted tail is closed with the typed recovery notice, and anything else is held. When the run that crashed compacted, its durable tail no longer extends the committed authority’s message prefix. The store then proves from its persisted rewrite rows that the physical head descends from the committed authority (a rewrite descendant), and recovery classifies the live tail after the last rewrite. Rewrite rows carry no run identity, so a rewrite descendant whose live tail holds no message of the run (an empty live tail) is held, and so is a tail whose rewrite descent the store cannot prove. Both fail resume with SESSION_DURABLE_TAIL_HELD_FOR_RECOVERY, never with internal-error prose, and recovery never closes such a tail on the store-issued run identity alone. Meerkat’s own checkpoints never write either shape without a run cursor: a checkpoint follows a model response, and the writer proves the descent before it writes. Since 0.9.5, on HeadCanonical, the first commit of the round after a compaction writes an empty live tail with the run’s cursor, and recovery continues the run from it as described above. Such a hold therefore points to a store changed outside Meerkat. The session’s committed history stays readable and its durable tail is kept intact. Keep a copy of the realm’s store for diagnosis, and continue the work in a new session. A committed strict descendant wins over a stale live runtime snapshot regardless of local actor liveness, and projection convergence requires intra-turn row provenance, so two committed sibling documents can never overwrite each other. Read-triggered recovery runs under an exclusive per-session fence, re-observes the head under it, and converges idempotently when a competing process wins the commit; a recovery that already landed is refused rather than committed twice. While a session is held or quarantined, its content is retained intact and resume fails typed rather than with internal-error prose:
  • SESSION_DURABLE_TAIL_HELD_FOR_RECOVERY - the durable tail exists but machine-authorized recovery has not promoted it yet; it awaits reconciliation. An operator resolves a tail held as ambiguous, or an interrupted tail held beside inputs another run left staged or queued, with rkat session resolve-held-tail.
  • SESSION_DURABLE_TAIL_RECOVERY_REFUSED - machine-authorized recovery was refused by conflicting persisted runtime facts (another live runtime, or boundary receipts that already cover or contradict the tail); retry after the conflicting runtime quiesces.
  • SESSION_DURABLE_EVIDENCE_QUARANTINED - the durable evidence is forked or unverifiable; the session is quarantined from resume.
All three carry the typed durable_resume_hold structured payload so callers can distinguish the hold class without parsing messages. Ownership of the pipeline (SessionDocumentMachine classifies, MeerkatMachine authorizes, RuntimeStore realizes) is documented in Machine Authority. A start or attach of a session whose durable tail is held for recovery or refused by recovery refuses the same way, before it reads the session: a turn, an input or event admission, a resume and a delivery into it. JSON-RPC answers -32003 and REST 409 SESSION_NOT_RUNNING, each with the hold’s durable_resume_hold payload, whose session_error_code names the hold; the MCP server’s meerkat_resume answers the same wire error. REST’s create and archive answer a hold the same way. Nothing is written and the tail stays for reconciliation; a continuation or job delivery into the session stays pending. A mob member restore answers MemberDurableTailUnprovable, as for a tail its recovery could not prove.

Boundary commits with no durable answer

Since 0.9.4 a persistent session’s boundary commit or run terminalization that reports no durable outcome within its store’s declared deadline marks the session reload-required. The deadline is RuntimeStore::commit_outcome_deadline: 120 seconds for the SQLite store (two 60-second busy windows). In 0.9.3 and earlier such a commit (a stalled disk or a hung network filesystem, for example) kept the session’s mutation gate for good, so every later wait for the gate, a mob destroy’s retirement of the member and a force cancel included, waited for it.
  • The runtime loop stops waiting and leaves the commit running; the commit keeps the session’s mutation gate until the store answers.
  • The run’s completion waiters fail with CompletionWaitError::DurableOutcomeUnknown, and a run whose boundary commit is rejected fails them with CompletionWaitError::BoundaryRolledBack. A JSON-RPC turn/start waiting on the run answers -32603 with data.reason completion_authority_unavailable. The input stays durably admitted: read it after the reload instead of resubmitting it. The public error class is still authority unavailable and only the message names the case, so resubmit only with an idempotency key.
  • Since 0.9.5 the session’s other completion waiters answer too: once the runtime loop exits, every waiter still on the registration (an input queued behind the run, for one) fails with CompletionWaitError::AuthorityUnavailable, which a JSON-RPC turn/start answers with -32603 and data.reason completion_authority_unavailable, and a message that the input stays durably admitted and runs once the runtime reloads. In 0.9.4 such a waiter got no answer until a reload or resume replaced the registration, and then only an untyped closed channel. A session that must reload for another reason (a resumed run whose actor the runtime cannot build, for one) answers its waiters the same way. Do not resubmit the input: read it after the reload.
  • Until the store answers, a reload refuses with the retryable RuntimeDriverError::RecoveryRepairBlocked; afterwards it replaces the registration from durable truth.
  • Once the session degrades, the waits for its mutation gate that serve live work end with RecoveryRepairBlocked instead of waiting for the holder: input admission, an exact-run interrupt and its reconciliation, the recover and destroy control commands, comms aborts, explicit stops, unregisters, closing an idle registration, member effects and the admitted-original read among them. Reads that take no gate answer without the held driver: whether the session has active inputs answers busy, an input’s terminal completion refuses with the retryable RuntimeDriverError::InputTerminalNotYetReadable (read it again after the reload), a stopped run’s contributor terminal is unknown, and the other reads refuse with RecoveryRepairBlocked.
  • A mob member’s tracked turn stays pending, as on AuthorityUnavailable. A scheduled delivery waits for the input’s own terminal outcome: it keeps its lease and re-reads that outcome for up to 5 lease periods (5 minutes at the default 60-second lease), then ends delivery_failed with runtime_completion_authority_unavailable, and the detail names the bound and the last cause. Nothing in that wait reloads the session (see Scheduling).
A graceful mob Shutdown that keeps queued inputs (QueuedInputsOnShutdown::Requeue on meerkat_mob::ShutdownOptions, and since 0.9.5 the default QueuedInputsOnShutdown::RequeueDurable for a member session that commits each round durably) fails the completion waiters of the inputs it keeps with CompletionWaitError::RequeuedAcrossStop. Such an input is neither failed nor consumed: it stays durably queued, and the session’s next start runs it, or ends it abandoned with reason requeue_expired when more than the Shutdown’s max age has passed since the Shutdown kept it, however long before the Shutdown it was admitted. Every start that still finds it queued (after a crash before it ran, say) checks that age again. Retiring, archiving or destroying the session before a start runs it ends it as any other queued input: abandoned with reason retired (retire, archive) or destroyed. Do not resubmit it; after the restart, the same idempotency key re-attaches to it. Only a durable prompt in the queue lane that is neither directed nor bound to an objective is kept: a member’s kickoff, and any other objective-bound input, ends as stopped, as without the option. The same stop, like an unregister with UnregisterQueuedInputs::Requeue, also keeps the run in flight of a session that commits every round: the turn stops at once, the run’s inputs stay bound to it with the same waiter answer and max age, and the run continues from its last committed round at the session’s next start (see Durable runs). A graceful shutdown that does not choose a disposition, a mob Shutdown included, uses this stop with a 15 minute max age for sessions that commit every round. A store fence gap answers the same. When a host store realm’s writer stays stopped through a run’s whole dispatch wait, or through the wait of a round commit that met the stopped writer, and no other process holds the fence, the runtime suspends the run with its inputs bound to it, and their waiters get RequeuedAcrossStop within that wait: the inputs run when the fence is re-acquired (the run resumes in the same process) or at the next start (an ephemeral input only in the same process). A run whose stop was asked (a cancel it accepted before the gap, an interrupt or a stop at the fence, or one of the suspended run) is suspended the same way and its waiters get the same answer, but it never resumes: once the fence is back the runtime settles it, its inputs abandoned as Cancelled. A waiter registered on such an input after the suspension gets the settlement’s answer, AuthorityUnavailable (“the suspended run was stopped before it resumed”). When another process holds the fence, however this process meets it (at a dispatch, at a commit or while a suspended run waits), the live session is torn down, the store keeps the session’s inputs as last committed for that owner, and every waiter in this process of an input the store keeps gets RequeuedAcrossStop too (see Durable runs). A run the gap suspended before its first round commit starts again from its inputs under the same run id, and publishes run_started again with no terminal in between. An observer reads a repeated run_started for a run id that has no terminal yet as that run starting over, and drops what it received for the run id before; a run that resumes from a round commit publishes run_resumed instead.
  • The generated wait-failure resolver classes such a wait: session not running, reason requeued_across_stop, resumable, and the surface’s pre-admission released. It answers the same after the session is gone.
  • A JSON-RPC turn/start waiting on it answers -32003 (session not running) with data.kind and data.reason requeued_across_stop, data.input_id, data.stopped_at and data.resumable.
  • REST answers 409 with code SESSION_NOT_RUNNING and the same details. It is not a retryable 503: an unkeyed retry after the restart would run the input twice.
  • A mob member’s bounded wait answers BoundedTurnFailure::RequeuedAcrossStop, and a same-key replay of an input that expired answers BoundedTurnFailure::RequeueExpired.
  • A scheduled delivery re-reads the input’s own outcome, as for the cases above.
An input admission whose store write never answers has the same deadline since 0.9.4. Past it the admission refuses with the retryable RuntimeDriverError::RecoveryRepairBlocked, naming the input, and the session needs a reload (runtime_admission_deadline), so the session’s next admission, its next run and a reload of it end typed instead of waiting for the store. The write keeps running and keeps the gate until the store answers, so the input may still be admitted: resubmit it with the same idempotency key, or read it after the reload. Sessions without durable state, and stores that declare no deadline, admit as before. Writes that no deadline covers still hold the session’s gate until the store answers: registration, unregister, control writes, recovery, the runtime loop’s writes other than a run’s boundary commit and terminalization (a run’s staging, for example), and the live-boundary work that follows an admission. A store declares the deadline through RuntimeSessionAuthorityOps::commit_outcome_deadline, so a pure decorator forwards it. A composed store whose commit runs lock-acquiring transactions of its own declares its inner store’s deadline plus each such transaction’s busy timeout. The default is DEFAULT_COMMIT_OUTCOME_DEADLINE; InMemoryRuntimeStore declares none.

Storage mechanics (schema ledger, fence, durability classes)

Every SQLite file in a realm carries a meerkat_schema(domain, version) migration ledger; stores bring their domain up to date at open under a pinned concurrent-open protocol, and a file whose ledger is ahead of the binary refuses typed (SchemaFromTheFuture) - a rollback candidate fails certification cleanly instead of crash-looping. Each database also has a sibling <file>.mfence lock file: store operations hold a shared per-operation guard, and offline maintenance (rkat storage migrate) takes the exclusive side after in-flight operations drain. A connection kept open between operations (a runtime delivery owner’s generation reader) also holds <file>.mpresence shared, and maintenance takes that lock exclusively after the fence lock, so it waits for the connection to close. Maintenance creates the presence file next to every database it fences, and rkat storage doctor inventories it with the fence lock file. Backup artifacts written by maintenance use the registered *.pre-<version>-<timestamp> naming that rkat storage doctor inventories and rkat storage prune owns. The eligibility check that runs at open, before a domain’s atomic migration, reads the schema ledger and the catalog in one snapshot (since 0.9.4), so a domain another process creates at the same moment is seen whole or not at all. 0.9.3 and earlier could refuse such an open once with “schema domain … has no ledger row but already owns objects”, most often when two processes made their first durable delivery write together. A durable delivery read decides which tables to read from the version its own check verified. Stored runtime input state is version 7 since 0.9.4, assigned by the generated persistence version authority: every input row a 0.9.4 binary writes is version 7, including a row it read at version 3 to 6. A 0.9.3 or older binary refuses a version 7 row before installing state and leaves the bytes in place.
Recent session-envelope upgrades are one-way once a newer binary writes the session. Before upgrading a durable realm, snapshot every durable store and keep that snapshot with the prior binary. Rolling the binary back after new writes can leave those sessions unreadable to the older release; a code-only rollback is not a supported recovery plan.
Every store slot a realm composes carries a machine-readable durability class (durable / rebuildable_cache / scratch) and its resolution. Declarations are complete by construction: a storage provider must declare exactly one class for each of the seven domains (sessions, runtime, schedule, workgraph, jobs, blobs, artifacts), so no slot dodges the rule by omission. A durable slot resolving to a non-persistent store without the realm manifest declaring that domain ephemeral is a startup error. A provider’s typed DeclaredEphemeral resolution is already an explicit declaration and does not need a duplicate manifest entry. Neither path permits a silent in-memory fallback. Deployment tooling can rely on the classes (for example, cloning only durable domains between state generations).

Session storage layout (0.7.25+): head-canonical strands

Since 0.7.25 the SQLite session store persists sessions incrementally: the transcript lives in append-only session_strand_messages rows, rewrite history in session_rewrites, and a single session_heads row per session carries the canonical metadata, message count, and CAS token. Saving a long session appends O(delta) rows instead of rewriting a monolithic document. The canonical-representation rule, per session: if a session_heads row exists, the head representation is canonical. A released 0.8.10 blob, if present, is consumed only by the one-time activation importer and is never a fallback authority for a current head. Sessions created in HeadCanonical mode may have no meaningful sessions blob row at all. Since 0.9.4 the store also lays out a whole-blob session whose transcript graph retired its oldest occurrences, which every session compacted more often than history_retained_rewrites (default 4) has. The realm’s one-time migration of such a blob row, the standalone WholeBlob conversion’s seeding and read-only head synthesis write or derive the rows the store keeps after retiring the same occurrences incrementally: the retired base as a session_transcript_retirements row and rewrite rows for the retained generations only, and the synthesized head is the head the migration writes. 0.9.3 and earlier refused such a session typed; the in-memory store still does.
Downstream tooling that reads sessions.session_json via direct SQL sees empty or stale content for sessions written by 0.7.25+ binaries, silently, because the column still exists. Raw table reads were never a supported surface; use one of the typed read paths instead:
  • SessionStore::load - full canonical session document.
  • SessionStore::load_meta - metadata-only projection (no transcript deserialization); list for filtered metadata.
  • The wire surfaces (session/read, session/list, REST equivalents).
For forensic tooling that must read the raw database, the canonical shape is: resolve the session_heads row for the session, then read session_strand_messages rows for (session_id, head.strand) ordered by seq over 0..head.message_count. Anything else is internal representation and may change without notice.

Range-read capability probes (load_canonical_head / load_rewrite_commits)

IncrementalSessionStore carries two additive read verbs that expose the head-canonical representation to future O(page) history reads:
  • load_canonical_head returns the persisted head row ONLY when head+rows are the session’s canonical durable representation. Unlike a full materializing load, it answers None for absent and non-canonical rows and never reads a WholeBlob body. Some promises the row is the persisted head itself and that load_messages over head.strand page-serves exactly the rows the head covers.
  • load_rewrite_commits returns the ADOPTED rewrite commits (idx < head.rewrite_count), oldest first, without materializing retained revision bodies. It must always equal load_rewrites’ commits, in order, including the empty set while a recorded rewrite is not yet adopted.
Both verbs have conservative defaults (None; derive from load_rewrites), so every existing store (including delegating wrappers that do not forward them) degrades to the whole-load path, never to a refusal. The conformance suite pins the contract conditionally: a store answering None for every canonical-head probe is fully conformant; a store answering Some must serve the persisted row and page-exact strand reads, and delegating wrappers must forward both verbs (capability_discovery chapter). Service reads trust only the store-selected profile and its issued authority. read_history, read_transcript_revision, and list_transcript_revisions may materialize the required domain view from the canonical rows, but a materialized Session never becomes write authority.

Durable runtime lifecycle vocabulary (runtime_states)

The runtime companion persists one lifecycle row per runtime. The full runtime_state vocabulary, with polling guidance: Busy-detection loops must treat stopped and destroyed as not-busy: stopped means nothing is executing and nothing will execute without an explicit resume, so polling it as “busy” hangs until an external timeout. Prefer the typed status surfaces over raw row reads: since 0.7.25, session/input_status answers from durable state without requiring a live runtime registration, so a fresh host process reports truthful terminal outcomes (Ok/NotFound) for work finished before a restart.

Config concurrency contract

Config runtime uses generation CAS.
  • config/get returns current generation.
  • config/set and config/patch can specify expected_generation.
  • Mismatches fail deterministically with generation conflict.

Data governance

No automatic sensitive-data redaction is applied to session content or tool payloads. Encrypt storage at rest externally if required by your environment.

Request-only turn context

transient_turn_context is non-empty runtime-input text, not Session state. Whitespace is significant and its exact text bytes remain on a pending input so cold recovery can retry the same turn, then retire with that terminal input payload. During the active logical turn, Meerkat projects the caller context as a distinct temporary user-channel injected-context message appended at the current foreground model-request tail at the authorized request boundary. Admitted steers never take this path: they are durable (see below). No conversational-row anchor is required. They are never System messages, transcript appends, rewrite records, compaction input, or extraction input. This is separate from durable injected_context, whose rows are appended before the turn’s conversational user message. Deferred creation cannot carry this context because no runtime input exists yet; attach it to the eventual turn instead. Browser/WASM sessions carry it through the same runtime-input path, with in-memory retention for the page lifetime. A Rust host that admits runtime input in its own process can also give a turn RuntimeTurnMetadata::request_only_context: request-only context that lives only in that process. It is projected exactly like transient_turn_context (an injected-context message at the tail of every foreground model request of the turn, never Session state), but it is never persisted: it is not serialized, so a stored input never holds it and no wire value carries it. It is not part of the prompt replay identity, so a retry of the same input with another value replays the original. An input recovered after a restart, and a run resumed from its ledger, carry none, so the turn recalls nothing rather than a stale value. A batch merge keeps each distinct value once, in order, and a durable steer that joins a running turn adds its own values to that turn’s later requests. A mob member’s work carries it (WorkSpec::with_request_only_context) only into a local session-backed member’s runtime input; remote, peer-only and autonomous inbox delivery refuse it.

Durable steer

A steer input admitted while its session’s turn is running reaches that turn at a cooperative model boundary: the point after the model returns tool calls and before the next model request. MeerkatMachine classifies every steer admission into one delivery class from the input’s typed structure, never from its content: Before 0.9.4 a peer message or request steer and a text-only prompt steer were request-only: the running turn saw them in exactly one model request and the transcript never recorded them. Every content steer is now durable or follow-up only, so a steer is never shown once and lost. A synthetic refresh-projection notice is one the next boundary’s notice refresh replaces: background_job and mcp_pending notices whose blocks are not persisted (for example shell job progress), and auth_reauth_required notices. Written into a running turn it would vanish from later requests, so it takes the follow-up turn. A background_job notice whose block is persisted (the durable completion record of a detached fork_off or council job) is not a refresh projection: it is durable history and eligible for in-turn delivery. Turn metadata that only a fresh turn applies is everything else a runtime turn start honours: System prompts, skill references, a tool overlay, additional instructions, request-local context, model, provider or auth routing, keep-alive, render metadata, a terminal peer-response intent and directed interaction IDs. A directed peer input or flow step owns a terminal its own turn must start, so it always takes the follow-up turn. A durable in-turn append joins a run that has already started: the run’s run-start hooks ran when it began and do not run again for the late rows (the same as for tool results). Incoming comms blocks among the appends publish the same peer_content_ingested events the follow-up path publishes. A durable steer that arrives while the model is streaming waits for the next boundary of the same run. If the run ends first (the model answered without another tool call), or the boundary refuses Session writes (structured-output extraction), the input takes the ordinary follow-up turn, which writes the same transcript rows. It keeps its admitted steer lane, so the runtime serves it first after the run, ahead of queued work. Exactly one of the two paths writes them:
  • The input joins the running run as a late contributor before its payload reaches the runner. Persistent runtimes make that binding durable first.
  • At the run terminal the runtime reads the typed delivery witness. An append that is in the session image the run commits or keeps is consumed with the run. An append the runner never wrote, or one whose image is discarded, returns the input to the head of its lane for one follow-up turn. A failed or cancelled run on a persistent session discards its image (the next turn resyncs from the store); an ephemeral session keeps it; a compaction rollback discards every append written after the rollback point.
  • boundary_append_applied is emitted when the runner writes the rows. It identifies the run and input, carries the model-projected content and append count, and can include the exact canonical notice rows and the first appended row’s transcript offset at application time.
  • Reconcile applications using boundary_appends_discarded and canonical transcript history. The discard event identifies the exact session, run, and input IDs whose applications were discarded, and is published after any required requeue persistence succeeds. A run terminal alone is not proof of discard. This event does not terminalize the input or invalidate a later application in another run; canonical retained history remains authoritative.
Durable steers are delivered one per boundary, in arrival order: a durable steer admitted while an earlier one still waits for its boundary joins the boundary after that one, in the same run. When the run ends first, or a boundary refuses the earlier one (for example a structured-output extraction boundary), the waiting steers keep their place in the steer lane and the follow-up turn delivers them in arrival order, ahead of queued work. Each steer is written exactly once. Arrival order holds within each delivery class: a follow-up-only steer (for example a directed peer message) admitted before a plain peer message still lands after it, because the plain one joins the running turn and the directed one takes the follow-up turn.

Withdrawing one queued input

From 0.9.15. MeerkatMachine::recall_queued_input(session_id, input_id) withdraws exactly one input, and only while no run has taken it. It is the counterpart of the calls below that end started work: a recall never interrupts a run.
  • The machine decides in one transition (AbandonQueuedInput) guarded on the input’s phase being Queued, under the session mutation gate and the session’s driver. A run takes an input on the same fact (StageForRun, and a durable steer’s join of a running turn), so exactly one of “a run takes the input” and “the recall withdraws it” commits, and concurrent recalls withdraw it once.
  • QueuedInputRecall::Abandoned { attempt_count }: the input was queued. It is now Abandoned with reason Cancelled (public terminal cancelled), committed durably on a persistent machine in the same commit as its runtime-termination receipt; its completion waiter is answered RuntimeTerminated. attempt_count is the machine’s count of run attempts for the input (a failed attempt that returned it to the queue); 0 means no attempt was counted, which is not proof that no run ever saw the input: a steer that joined a running turn and went back to the queue because the turn discarded what it had appended counts no attempt.
  • QueuedInputRecall::NotQueued { phase, run_id }: a run holds the input (Staged, Applied or AppliedPendingConsumption). Nothing changes. run_id is the run that holds it, when the runtime has one.
  • The list of outcomes can grow (QueuedInputRecall is non-exhaustive): keep an outcome you do not know as it is.
  • QueuedInputRecall::AlreadyTerminal { terminal }: the input already ended. Nothing changes. A second recall answers this, from the live ledger or, once the terminal row is archived and after a restart, from the store.
  • Ok(None): the session holds no such input, an input of another session included, as the per-input reads answer.
  • The call never loads, wakes or attaches a session. With no registration in the process, a terminal stored row still answers AlreadyTerminal; a row that is not terminal is refused NotReady (its phase lives in a machine that is not loaded), and an input a graceful stop kept queued is recalled once the session serves again. A machine without a store answers from its live registration only, and refuses a session it has not registered as NotReady.
  • The recall is session-level. Any caller that can act on the session can recall any queued input of it, whoever submitted it; on the wire also by an idempotency key the caller did not mint. Recalling an input withdraws everything the queue coalesced into it: an earlier input that a later one took in is already terminal (Coalesced, naming the later input), and a recall of the later input withdraws both contents. The answer names only the recalled input; it does not say that other content went with it.
  • An error is not an outcome. When the durable commit of the abandon fails, or a step after it does, the call returns that error, the session needs a reload, and it refuses a recall until then. The store decides what happened: after the reload (or a restart) the input is either queued again, and can run or be recalled, or terminal and never taken by a run. No call reloads the session: the runtime reloads it from its store the next time it prepares the session for a turn, and at a restart. So after a durability refusal, start the session’s next turn or restart, then read the input’s status. Until the reload the per-input read answers the live machine’s row, which can be ahead of the store (the abandon applied, its commit failed): it is not the recall’s answer.
  • A recall can land while the submit of the same input is still preparing to hand it to a running turn. The recall wins: the input is withdrawn and nothing is delivered, and that submit call can answer an error in place of its admission.
On the wire this is session/recall_input (JSON-RPC, with the input named by id or by idempotency key) and the SDK wrappers; there is no REST route and no mob-level route to a member on another host.

Stopping one run

MeerkatMachine::stop_run(session_id, expected_run_id, reason) stops exactly one run and every input already bound to it. The exact-input cancel_input_if_present and the exact-run hard_cancel_run_if_current only interrupt the run, so a durable steer that joined it still follows the rules above and can take a follow-up turn. Since 0.9.4 cancel_input_if_present and cancel_tracked_input_for_member_incarnation do not poll an input caught between lifecycle steps (neither queued nor terminal, with no run yet). They wait for that input’s next committed lifecycle step (its run association, a phase step or its terminal), which the runtime driver signals at its single commit point, and read the input again then. The 5-second settle bound is only a hang guard; a cancellation reads the input once more before it reports that bound. 0.9.3 and earlier re-read such an input every 25 ms.
  • Under the session mutation gate, the machine compares its current run with expected_run_id and records the stop (StopCurrentRunForRun). It then dispatches the exact-run hard interrupt.
  • From that point the run admits no durable join. A steer that reaches its boundary later is not a contributor of the stopped run. It stays queued and runs afterwards like any other queued input.
  • At the run terminal, a joined durable steer whose append is not in the surviving image is abandoned as Cancelled instead of returning to its lane. A retained append is consumed with the run. A failed attempt never replays its staged batch.
  • The call returns RunStopReceipt::Stopped { run_id, contributors } once every contributor staged at the stop has reached its canonical terminal. Each entry carries the delivered completion and the committed input terminal. A cancelled join’s completion is RuntimeTerminated, while its committed terminal is Abandoned { reason: Cancelled }, so read the terminal for the lifecycle fact.
  • Once the stop is committed, an interrupt dispatch that fails, times out, or finds the executor not yet inside the run does not fail the call. The run still ends as a stopped run, and the call returns at its terminal.
  • A stop whose run is no longer current returns RunStopReceipt::NotCurrent { run_id, current_run_id } and touches nothing. Queued input and newer runs are never interrupted.
  • If a runtime stop or teardown has taken the still-bound run between the compare and the stop, the call returns RunStopReceipt::NotStoppable { run_id, state } and stages nothing.
  • A run a store fence gap suspended (or a restart marked resumable and has not resumed yet), and a run whose session store’s writer is stopped, are stopped without waiting: the stop is recorded at once (the latter run is suspended, not interrupted), and the call returns RunStopReceipt::Stopped { run_id, contributors } with no contributors. The run never resumes. Its inputs end Abandoned { reason: Cancelled } when the runtime settles it (at its loop’s next lap for a restart’s run, once the session’s store takes writes again for a gap’s): read their outcome later. A process that ends first, or another writer that takes the session over, runs them instead. A graceful requeue stop, or a Keep rollback, that begins before that settlement ends them Abandoned { reason: Stopped } and leaves the run to the next start, which repairs its tail as an interrupted run (no run terminal is published for it). A stop that races the gap’s own suspension of the run can fail with an internal error naming an input whose wait was answered RequeuedAcrossStop. Where the run cannot be suspended (an executor that cannot suspend, a stop that joins a hard interrupt already in flight, or a published live-boundary join not yet retained), the stop is a hard interrupt, which the gap undoes with a reload that resumes the run.
The stop intent lives in machine state and is not recovered after a crash. A process that dies mid-stop recovers the run’s staged inputs like any other interrupted run.

See also