Mobs are the multi-agent path in Meerkat. The agent-facing
delegate tool,
explicit mob_* tools, SDK Mob classes, mob/* RPC methods, public MCP mob
tools, and mobpack deployment all route through the mob system.Choose A Path
rkat run keeps its sessions in memory, so the members of a team an agent
creates there end with that run’s process. A team that must outlive the
process belongs on a host that keeps sessions.
Mental Model
AMobDefinition describes profiles, limits, wiring rules, topology, and
flows. The running mob records the roster, member lifecycle, events, flow runs,
and work state. Each member is an agent session, but public mob APIs address the
member by stable AgentIdentity, not by an internal session or runtime binding.
Core Concepts
Agent Tools Vs Host APIs
Keep this distinction sharp:
Agent-side tools are late-bound through the session build path. Host APIs are
the stable control plane for applications and SDKs. Do not treat raw
mob_*
agent tools as if they were JSON-RPC methods.
Fast Path: Delegate
delegate creates an implicit session-owned mob on first use, spawns a helper,
and wires the helper to the creating session. Use it for bounded helper work
that should report back.
implicit_delegate_idle_retire_secs (300 seconds by default).
Unless its tooling supplies a profile, a helper runs on the model and
provider (and self-hosted server) the calling session runs on when the call
runs, so a session on a realm [models.<id>] entry or a self-hosted model
delegates. That is the session’s current identity: after a turn that named
another model, provider, self_hosted_server_id or auth_binding, the
next delegate follows the switch. The helper also resolves its credentials
through the session’s auth binding; a session with none leaves the helper to
the realm’s resolution. Three helpers do not take the session’s binding: one
whose tooling supplies an inline profile builds on that profile, which names
no binding; one whose tooling names a realm profile builds on that profile’s
binding, or on the realm’s default; and one placed on another host resolves
its credentials on that host. A delegate whose session’s route changed since
the last one replaces
the session’s implicit mob, with everything in it (members spawned there
included), and its result says it created the mob. The current identity is
read from the session’s stored metadata, which rkat and the JSON-RPC, REST
and MCP servers keep; a session on the in-memory ephemeral session service,
which keeps none, delegates on the route it was built with.
A session’s implicit mob (its delegates and the forks seated there) and the
mobs its agent created belong to the session, and they end with it: they are
destroyed when the session is archived. From 0.9.15, besides that, only the
route change described above replaces the implicit mob, and a mob the agent
created ends when it is destroyed by name: a turn that fails, is cancelled
or is abandoned, a request that is cancelled, and a stop of the host end
none of them, and the session keeps its mobs for its next turn.
For recurring teams, use explicit mobs instead of a chain of ad hoc delegates.
A mob member can also fork itself with fork_off. The child starts from the
member’s exact committed transcript, runs one task in the background, and
belongs to the forker, which can check, list, and retire it (and whatever it
forks in turn) without manage scope over the mob. The result is recorded in the
forker’s transcript as a durable BackgroundJob system notice: a forker still
in its turn sees it at the next model call, and an idle forker gets one wake
turn; on a one-shot rkat run without --keep-alive the
call blocks for it instead. There is no default deadline; max_run_secs is an
opt-in autokill. See Mob tools
for the result shapes and the ownership rules.
A fork child holds the instruction activations it copied from its source as
history only, and runs no model turn until each inherited key is activated in
the child. The host decides who does that with its fork instruction policy
(MobBuilder::with_fork_instruction_policy, or
MobMcpState::with_fork_instruction_policy); it is never a tool argument or
part of a mob definition:
require_explicit_activation(the default): the host activates the child afterfork_member. The forks that start the child’s turn themselves (fork_off, the bounded and detached member forks, the live delegation’s durable fork, and council participants) leave no window for that, so a fork of a member with activated host instructions is refused before anything is committed, with the typedfork_instruction_refusederror.reactivate_inherited: the fork appends the child’s own activation of every key that was effective in the source, and the child runs at once. A fork (durable, orfork_helperbelow) whose carried head of a key is a row whose body a compaction of the source retired (a later row of that key, past the fork point, superseded it) is refused with the typed causehead_body_retiredbefore anything is committed; a fork at a later boundary carries the superseding row with its body.refuse: every fork whose child would inherit an activation is refused.
load_skill, or was
given as per-turn skill context, follow the same policy as the host set it:
under the default each copied skill is named without its body, under
refuse the fork is refused, and under reactivate_inherited they are
carried. A fork that leaves no activation window applies this as set, so
under the default it strips the skills instead of refusing.
fork_helper (a member spawned with MemberLaunchMode::Fork) starts a
fresh session instead and gives it the source’s recent conversation as
context. The same policy decides what of it the child gets, by each row’s
type. The source’s system prompt and host system context are never
carried. Under the default, the source’s activated instructions, the
skills it loaded with load_skill (each named in their place) and
host-injected context such as memory recall are left out; under refuse
a fork whose context holds an activation or a loaded skill is refused;
under reactivate_inherited they are carried. A peer message in that
conversation keeps its sender, its text and its sender’s taint marker; a
peer request or response keeps its sender and a preview of its params or
result (a response also its status). Neither keeps the ids or instructions
to answer it: the child is not its addressee.
The context fits a byte budget, newest rows first, and a fork whose newest
row alone does not fit is refused with fork_context_too_large.
A fork child is told what it was given. Before its first input it gets a
fork notice (a fork_boundary system notice) that names its source and,
by name only, the instructions and tools it lost or gained and what its
context carried. A fork_helper child gets its context the same way, as
one row before its first input, so a turn-driven helper sees it in its
task turn. A child placed on a member host gets them the same way when
its host serves prompt-context fork children (a host of this release
does); a host of 0.9.4 or earlier gives it the context and the notice as
the start of its first prompt instead.
Define A Mob
A small mob definition has profiles and optional wiring:council-- is reserved: it is the id
form of the
mob a temporary council
runs in. A create under such an id is refused, on every surface, as an
invalid definition (reserved_system_identifier at id). A mob an earlier
release created under such an id keeps loading and working; the mob state
logs one warning that names it.
Read-only profiles
tools.read_only = true declares a profile whose members may execute only
tools their owning dispatcher declares read-only. It is enforcement at the
execution gate, not prompt guidance, and it is the floor: a spawn request
cannot widen it.
access_denied tool error, so the run
continues and the transcript records the refusal. The model-visible tool list is
unchanged (gating never moves the prompt-cache prefix).
What the declaration covers, and what it cannot:
Anything undeclared is denied: over-denial is honest, silent admission would be
a false guarantee. Hosts widen the read surface by implementing
AgentToolDispatcher::tool_mutation_class on their own dispatcher.
Denying individual tools
tools.deny names tools a profile’s members may not run even though their
family is enabled, for example the topology tools of mob = true:
read_only, a deny list works by name, so denied tools are hidden from
the member’s tool catalog (the tool list sent to the model) on every build, a
resume included. A call to a denied name is still refused with an ordinary
access_denied tool error, and a spawn cannot widen the list. An unknown name,
such as a typo, fails the member’s build with DeclaredToolUnknown, unless the
profile declares an MCP server that maps no tool_names or opts into a bundle
the build withholds; then the name is checked only at call time.
Granting tools to a member at run time
tools.grant_gated mounts built-in tool families, tool names and the tools of
MCP servers hidden: the member is not shown them and every call is refused
until a live scoped tool grant opens
them, for a bounded window. Leave a gated family’s switch off; the gate mounts
it closed, and a binary that does not know the gate keeps it off.
tools.mcp_servers) starts no
process until a live grant of that server ({"kind": "mcp_server", "server": "tickets"}) covers the member at a build, so the member connects it at its
next build, a respawn for one, and not before; the build decides this as it
composes the member’s tools. unconfigured_mcp_servers = true gates every MCP server the profile does not configure, whatever path
attaches it later; the servers it configures stay open.
At every build, resume included, a member’s gate is its own profile’s gate
joined with its role’s current gate, so a member spawned with its own profile
still gets its role’s gate at its next build. A store-owned identity declared
with its role’s named profile is built from the role’s definition, as a worker
of the role is: its own gate is the role’s current gate, less the families its
declaration switches off; one declared with an exact profile of its own
(profile_override) gets the role’s gate as role entries, and so does one
whose desired material does not record where its profile came from (one
adopted by an earlier release). A role entry never fails the build: a role
tool name and a role MCP server are gated from the moment they appear, a role
family is gated where the member mounts it, and a role family the member does
not mount is ignored. A grant to the member’s identity (tool_grants/grant
with {"kind": "identity", "mob_id", "identity"}) follows the member across
respawns and restarts, and ends when the identity ends, with the grants of
the session the mob holds for it then: at the end of a plain retire; at the
end of a retirement that unlists or removes an identity entry; at the
unlist of an unseated identity entry, or a complete roster’s seal of one it
does not list and no host’s intent row names; at the removal
of an unseated store-owned identity; when a respawn whose replacement fails
after the old member’s archive is abandoned, by its own step or by the
resume after a stop; or at a reset of the mob. A respawn keeps them, and so
does a host’s adoption of a declared identity, which ends the identity’s
entry, not the identity. A name
created again needs a grant of its own, and a later resume, which replays
the earlier respawn’s abandonment, ends none of its grants. A grant between a respawn’s
retirement of the old member and the seating of its replacement answers
subject_unknown, since the identity is out of the roster then: retry it
once the respawn answers. An identity grant answers applies,
whether the member’s running build gates its target (at_next_build for a
gated server the profile configures that the running build left
unattached), and tool_grants/list
answers the member’s gate, running and next build. A member whose gate names
anything cannot be placed on another host.
Opting into host tool bundles
tools.bundles names host tool bundles a profile opts into. It is the lenient
twin of the host-named rust_bundles: a member build refuses with
ToolBundleUnavailable while a rust_bundles entry is not registered, but it
mounts each bundles entry the mob’s builder registered and withholds the
rest. MobHandle::withheld_tool_bundles() reports what is withheld. The
registrations are the ceiling and are read again at every build, a resume
included, so a host that stops offering a bundle withdraws it from every
member that opted into it. While any opted bundle is withheld, every deny name
in no vocabulary is checked only at call time, so a typo is reported at build
only once the bundle is mounted again.
Through the public profile input (the agent mob_create tool, mob/create),
only an inline profile of a child mob, a mob a member creates with
mob_create, may opt in, and only to a bundle the host registered as
ChildToolBundleAvailability::ChildOffered. Any other name is refused with
ToolBundleNotOffered, which lists the bundles the host offers; an empty or
repeated name is an argument error. Realm profile writes and spawn-time
profiles given as public profile input (the agent mob tools and the MCP
profile tools) cannot opt in. The
offer is the host’s, the same for every child mob, whatever tools the creating
member itself mounts. A bundle the host registered as ChildAvailable still
reaches every inline child profile without an opt-in.
Profile model and provider fields
Beyondmodel, a profile can pin provider identity and per-member behavior:
provider_params = { reasoning_effort = "high" }. The
member’s build checks the level against the profile’s model and route like
any session’s, and refuses it with the same typed reasoning_effort_refused
data. From 0.9.15, a member built from such a profile holds the level
(also when the profile came with mob/spawn’s override_profile): while it has a
turn running or queued, a JSON-RPC request that changes its model,
provider, provider params, auth binding or level is answered
SESSION_BUSY at once and is not queued (see
Model settings on a busy session).
The request is mob/turn_start for a turn_driven member, and
turn/start with the member’s session id for an autonomous one
(mob/turn_start is refused for an autonomous member with -32602). An
autonomous member runs turns of its own, so the caller cannot know when
it is idle: send the request again, or set the level in the profile or
at the spawn.
No flow step, kickoff, peer message, member send, scheduled delivery or
wake is answered busy; the rule is on turn/start and mob/turn_start.
A resumed member keeps the level its session recorded. With
provider_params in resume_overrides, a level the profile’s params name is
seated with them, and params that name no level leave the recorded level in
place; add reasoning_effort to resume_overrides to make the profile
decide the level on resume in every case: the level its provider_params
name, or none. A member host that runs 0.9.14 refuses
a member spec that carries a level: it fails closed with a spec digest
mismatch and does not ignore the level, so upgrade member hosts before a
profile sets a level.
From 0.9.15, a level in a profile is a one-way step for a rollback. A mob
whose definition holds a level in any profile, even a profile no member ever
used, cannot be read by 0.9.14: after a rollback 0.9.14 quarantines the whole
mob (its content is kept, and the mob is whole again on 0.9.15), and a Meta
gateway on 0.9.14 does not start on a state directory that holds such a mob,
so nothing stored in that directory can be used there. Removing the level
from the definition afterwards does not help; the only way back is to
destroy the mob on 0.9.15 and create it anew, which loses its members,
their conversations and its history. See
Rolling back to 0.9.14.
If the route stops accepting the profile’s level between a member’s spawn
and its resume (an account’s model list changed, or the model was swapped),
only that member is affected. Under on-demand restore it stays dormant until
its first wake and is then listed as broken with the reason; under eager
restore it is broken at start. The mob keeps running and nothing is lost: the
member keeps its seat, its session and its conversation. To recover, make
the route accept the level again or change the level, then respawn that
member (the same session, the conversation kept) or restart the host. A
plain retry does not seat it, and removing the level from the profile alone
does not help, because a resumed member keeps the level its session
recorded.
A profile’s auth_binding is an explicit build input: its members build on
that binding, a resumed member included, and a spawn that names its own
binding wins over it. Only an operator’s own definition may name it: a mob
pack whose definition names one fails validation, and Connect refuses an
application setup definition that names one. An owner can also write a realm
profile that names one, through JSON-RPC mob/profile/create and
mob/profile/update and the MCP profile tools; reading that profile back does
not show the binding. A member placed on a member host
resolves its credentials there and does not carry the profile’s binding, and
an adaptive layer refuses a profile that names one.
Otherwise, a resumed member keeps its durable credential binding while its
provider is unchanged. When provider in resume_overrides moves a member to
another provider, the member keeps its history and resolves the new provider’s
binding from its realm, as a fresh member of the profile would: the realm’s
binding for that provider, then the parent realms and global, then the
provider’s environment key. A durable binding backed by a credential account
keeps that account; if no binding for the new provider shares it, the resume is
refused with a typed error and the durable session is left as it was. A host
that runs a member for another host needs the release with this behaviour to
resume it.
Anthropic prompt caching is a per-profile provider_params decision. From
0.8.33 the default is automatic on every backend that supports it (Anthropic
API, Vertex, and Foundry); Bedrock and other backends that report automatic
caching as unsupported stay disabled. Releases 0.8.22 through 0.8.32 default
to disabled on every backend, so a mob that must behave the same on both
sides of that boundary sets cache_control explicitly. Opt a profile out, or
force caching on, with the nested provider_tag form:
cache_control lives inside provider_tag, not at the top of
provider_params. The flat form provider_params = { cache_control = "disabled" }
is rejected by the closed field set of ProviderParamsOverride and fails the
whole definition parse, so a mob that boots today stops booting after that
edit. The vocabulary is automatic, system_prefix,
system_and_conversation, and disabled; cache_ttl is 5m or 1h and
must be paired with an explicit non-disabled cache_control in the same tag.
Add provider_params to resume_overrides when the change must reach members
whose durable sessions already exist.
Member system prompt
peer_description is the roster blurb other members see. It becomes the
member’s PeerMeta.description, so it shows up in a peer’s peers tool output
and in the mob.peer_added lifecycle notice wired peers receive. It is never
part of this member’s own system prompt.
The member’s own prompt comes from profile.skills. Each listed id is resolved
against a [skills.<id>] table in the definition, and the resolved texts are
joined in list order into the system prompt. An id with no [skills.<id>]
table is a missing_skill_ref validation error and contributes nothing:
DurableAgentSpec.additional_instructions,
appended after the skills text and baked once at materialization, and an
AgentCustomizer whose draft.system_prompt replaces the assembled prompt.
A host that keeps its own per-profile prompt key (for example a role_summary key)
parses that key itself and lowers it to additional_instructions.
A member’s session records the instruction sections it was built with in its
durable build state, and keeps them through every rebuild: a restart, a wake
and a fork child’s seating resume the session, name no sections and write no
prompt, so they leave the record alone (a fork child records its source’s
sections). A Rust host reads them with
MobSessionService::recorded_build_instructions(session_id) on the mob’s
session service, for the member’s session
(MobHandle::resolve_bridge_session_id). The read comes from the session’s
committed state, without waking the member or decoding its transcript:
A seat no build has created yet has no session, so the read answers
None
for it: its sections are still its seat’s spec. Its first build may append
the sections the host’s spawn customizer adds, so afterwards the session
records the spec’s sections followed by those.
The read has limits. It answers for the service’s own sessions, so a session
another host keeps answers None. It takes no caller principal: a surface
that serves it admits its caller first. On a persistent session service it
still answers an archived session’s record.
PersistentSessionService answers from the session’s committed metadata
(the committed head on a HeadCanonical store; on a WholeBlob store the
committed body is fetched and verified, and its rows are skipped), and Core’s
RPC and CLI hosts forward the read to it. A host whose session service is
its own decorator forwards the method to the service it wraps: the trait
default answers the same for a session that is not archived, from the
persisted session, which loads its transcript.
There is no system_prompt key on a profile. Loading a TOML definition whose
[profiles.<name>] table carries system_prompt (or its aliases prompt and
instructions) fails with MobError::UnsupportedProfileKey, whose message
names the profile, the key, and the mechanisms above; a realm-reference table
(realm_profile = "...") is refused the same way. Any other key the table
does not declare (a host-private key such as a downstream app’s role_summary, a typo
such as comm = true under [profiles.<name>.tools], or anything but
realm_profile on a realm reference) is reported once per key as an
unknown_profile_key warning diagnostic, returned by
MobDefinition::parse_toml, and from_toml logs one warning per affected
profile when the definition loads; parsing continues, and the key is ignored.
No Meerkat surface emits that diagnostic today: a host that loads mob.toml
itself calls parse_toml to surface it, and a definition supplied as JSON is
not inspected.
Custom model registry entries
[models.<id>] tables declare uncatalogued models once, at the definition
level. One entry feeds provider inference, compaction scaling, capability
gates, and call timeouts for every profile that references the model:
[models.<id>] table shape works in the host’s config.toml under
[models], next to the per-provider default model strings.
At load time, rkat mob validate rejects a profile model that is neither
catalogued, custom-defined under [models.<id>], nor provider-annotated
(unknown_model), instead of failing at the member’s first delivery.
Create And Spawn
- JSON-RPC
- Python
- TypeScript
- Web SDK
- CLI
Autonomous members run as long-lived peers.
turn_driven members are useful
when a host wants explicit dispatch control: only they take mob/turn_start.
A fresh autonomous_host member runs exactly one kickoff turn (one model
request) when it is spawned: on its initial_message, or without one on a
fallback role prompt. A member that must start idle should be turn_driven; it
runs initial_message as its first turn, and nothing without one.
A resume of an existing session (a resume launch mode) delivers that first
turn only while the session’s first turn has never started, and then once.
Otherwise the resume runs no kickoff and does not deliver initial_message: the
transcript already holds the member’s start, and a first turn that started and
was interrupted is not re-run. A placed member (placement set) follows the
same rule: its member host reports whether the adopted session’s first turn
started. A member host of 0.9.4 or earlier reports nothing, and a resume there
still runs the first turn.
mob/ensure_member (Rust MobHandle::ensure_member) is the idempotent spawn:
it spawns the member only when its identity holds none and otherwise answers
the existing member, and mob/ingress_interaction ensures its ingress member
the same way before it delivers. When a spawn of the same member is still in
flight, the call waits for that spawn to seat the member and answers with it,
within a 60 second ready-wait budget, after which it answers
ReadyWaitTimedOut; if that spawn fails before its member is committed, the
waiting call spawns the member itself. A second spawn of a member whose first
spawn has not settled yet is refused MemberAlreadyExists.
An ensure of a reserved name (a removed identity’s, or a name this build
cannot read) is refused IdentityMember, as a spawn of that name is, before
anything is recorded or created.
Trusted Rust member build overrides
An in-process Rust host can attach a per-memberArc<dyn CompactionCurator> through
SpawnMemberSpec.compaction_curator_override. The exact curator reaches the
session build for that spawn or cold resume. It is executable process-local
behavior, not durable configuration: re-supply it from a
SpawnMemberCustomizer after restart. Remote placement rejects it before
materialization instead of dropping the override.
A host gates a member’s tool calls with a per-member
SpawnMemberSpec.tool_dispatch_admission (an Arc<dyn ToolDispatchAdmission>),
usually set by its SpawnMemberCustomizer. It is process-local: it is never
persisted and never copied to a fork. When a member whose build runs under one
is forked (the agent tools fork_off and council, MobHandle::fork_member
and its siblings, a live delegation), the customizer is asked for the child
before the child is seated, and only the tool_dispatch_admission of its
answer is used: the child runs under it, or the fork is refused with
MobError::DispatchAdmissionRequired. A refused fork_off or handle fork
forks nothing; a refused council branch is refused after the council forked
its source for it, and the council’s cleanup removes that fork. A
fork_helper child of such a member is asked as any spawn is, and is refused
the same way when the customizer installs none. Every later build of such a
child (a wake, a restore after a restart, a reload, an explicit resume) and a
respawn successor of it ask again, and are refused when the customizer
installs none, so the child never runs without an admission. A fork child
seated before 0.9.11, whose spawn row records nothing, is asked the same way
each time it is rebuilt and runs under the admission the customizer installs,
or under none when it installs none. A member a spawn policy seats is asked
too, for its admission alone (SpawnSource::PolicySpawn), and runs under what
the customizer installs. Before 0.9.11 such members ran with no admission.
A trusted host that decides each member’s destination or agent mandate
installs a MemberRuntimeRegistrationHook with
MobBuilder::with_member_runtime_registration_hook (runtime-adapter). Since
0.9.4 each call carries the member’s exact registration
(MemberRuntimeBuildIdentity::registration()): the one prepared for a new
member, or the current one of a member whose actor is recovered. Bind what the
host decides to that registration, not to an identity derived from the
member’s session id or labels; a later registration of the same session is a
different one. The identity is no longer UnwindSafe, so wrap it in
std::panic::AssertUnwindSafe to move it into catch_unwind.
Since 0.9.4 a hook that refuses a restored member’s registration
(select_destination, select_agent_mandate, prepare_member_surface or
verify_member_surface) gets the same cleanup as a refusal for a new member:
the restore removes that registration exactly (recorded in the session’s
history; inputs it still held end Stopped) and releases its owner context.
The member stays Broken with MobError::MemberRegistrationRefused, shown as
registration_refusal in its status, until an explicit repair, and
ensure_member_live returns that refusal without asking the hook again. The
member can still be retired, respawned or destroyed. A member that
MobBuilder::resume recreates on a fresh session (for example when the session
service keeps no durable sessions) is isolated the same way when its hook
refuses it, and the other members resume. A destination the hook
selected that the runtime did not install is
MobError::MemberDestinationNotInstalled instead (the runtime’s verdict, not
a refusal): the member keeps its registration, and the repair must select
again with an advanced context_generation. Since 0.9.9 the hooks are called
as tasks on the host’s runtime, where the member’s turns run, and a call that
ends without an answer is not recorded as the host’s refusal.
Since 0.9.9 every member build a mob runtime performs (a fresh spawn, a
restore or a wake) carries that runtime’s incarnation witness in
SessionBuildOptions::mob_incarnation (AgentBuildConfig::mob_incarnation),
so a host’s member build hook mints the member’s mob operator scope for the
mob as it runs, also for a member restored at boot before the host holds the
mob’s handle. The witness is never persisted, and a placed member built on a
host that does not run its mob carries none. The mob runtime also receives the
live mob operator authority of every member session it builds locally:
MobHandle::member_spawn_profile_scope_contains(origin_session, profile) and
MobHandle::member_manage_scope_contains(origin_session) read whether that
session’s own authority holds spawn-profile scope for the profile, and manage
scope, in the mob as it runs (the facts the spawn tools pass to the mob
machine’s spawn admission). Each reads only that session’s authority and
finds none for a session the runtime did not build or whose build bound none.
A hook that edits a member build’s external tools wraps them: tools it
replaces outright leave the session with no recorded authority.
Identity And Respawn
Mobs separate stable member identity from runtime binding details:
Use
AgentIdentity for facts that survive respawn, such as wiring and durable
configuration. Runtime IDs and fence tokens protect the lower-level binding.
Since 0.9.4 a spawn never reuses a generation, across mob resets and restarts
too: resume recovers each identity’s highest generation from the whole event
log, so a member re-spawned after a reset and a restart never reuses its
pre-reset runtime id or member address, and a mob that was reset and then
re-spawned a member under a pre-reset identity resumes after a restart (0.9.3
and earlier failed that resume with an internal error). A store written
before 0.9.4 whose last epoch already reused a generation still resumes, and
that identity’s next spawn goes above the recovered high-water.
When a session’s mobs end depends on what archives the session. A member
its mob retires has its session archived by the mob, and the mobs that
session owns (the mobs its agent created with mob_create, and the session’s
implicit mob with its delegates and forks) end with it at the next start of
the host: the mob’s retire archives the session and leaves its mobs to that
start. The archive of a store-owned identity is such a retirement: each of
the mobs its agent created is removed with its store at the host’s next
start, and unarchiving the identity does not bring them back. A session
archived through a surface’s own archive call (JSON-RPC
session/archive, the REST and MCP archive) has its mobs destroyed at that
call. A
worker’s respawn (respawn, mob/respawn, the mob_respawn tool of
MobMcpDispatcher) retires the member and gives its successor a new session,
so the mobs the old incarnation created and its delegates are not carried
over to the successor.
An identity entry’s member is respawned on its own session
(MobHandle::respawn_member, and respawn of such a member), so the mobs
that session owns do not end with the respawn, and its agent keeps managing
the mobs it created with mob_create: that management follows a member
through its same-session successors, and ends when the member is reset onto
a fresh session or is retired and later seated again. Which side a member
is on follows from how it was created: a member a
host seats through an identity entry (a declared or store-owned identity, the
way a host keeps a durable roster) is an entry’s member; a member spawned
directly (a host’s spawn call, an agent’s mob_spawn_member, a helper) is a
worker.
Dormant seats and comms keys (Rust hosts)
A dormant seat (MobHandle::seat_dormant_members, a rotate_member that
seats, an identity entry’s seat) records a member’s comms endpoint before the
member’s first build, so it reads or mints the member’s key pair where that
build will load it. On in-process comms, when the host declares no root, the
seat takes the root the mob’s session service reports
(session_comms_identity_root: where its agent factory keeps each session’s
key pair, under the factory’s user_config_root; see
where a session’s keys are kept), so a host that composes
its own session service needs no
MobBuilder::with_member_comms_identity_root. A declared root, and
MobBuilder::with_network_member_comms, win over the report. A service that
reports NotKept, or a root it cannot resolve, has every seat refused typed
(MobError::WiringError, naming the report) unless the host declares a root:
no other root stands in for the one its builds use. The report covers
in-process keys only. A host whose members run on TCP or UDS declares
MobBuilder::with_network_member_comms; without it the seat records an
in-process endpoint, the member’s first wake refuses it as an endpoint
mismatch, and the member needs a rotation (MobHandle::rotate_member).
Durable role migration
A durable member may keep the same mob id,AgentIdentity, session, and
transcript while moving to a new role. This is a one-request cold Resume
contract, not a profile alias or standing migration rule.
For an explicit trusted Rust-host resume, declare the exact durable predecessor
role on the spec:
SpawnMemberCustomizer and call spec.declare_resume_from_role("domain")?
only when ctx.spawn_source == SpawnSource::Resume, after checking the exact
member identity and requested target profile. The declaration is intentionally
not persisted, so each later migration or restore must re-supply current host
intent. A same-session respawn that names another profile
(MobHandle::respawn_member with MemberRespawn::with_profile) declares its
migration itself: the next incarnation’s spawn row records the role its
session was last built under, so its first build restamps the session, after a
restart too, with no customizer (see
Respawn, rotation and scoped release).
The runtime enforces all of these facts:
- mob id,
AgentIdentity, and exact durable session stay fixed - the declaration must equal the one stored predecessor role
- when the stored and requested roles differ, an omitted declaration returns
MemberRoleMigrationRequired - an incorrect, inapplicable, or live-session declaration returns
MemberRoleMigrationRejected - even an idle live actor cannot be migrated; the exact session must be cold
- success restamps the current comms name, typed member binding, role/profile labels, callback context, and explicitly configured tooling
- rollback is another declared forward migration or separately versioned durable state, never an old binary silently reclaiming the predecessor role
Declared identities (Rust hosts)
A Rust host that keeps its own roster of durable identities declares it, and Core applies it:MobHandle::apply_declared_roster. Pass the host’s whole
roster as a complete one, and apply it once after every start, before the
host wakes or seats its identities:
DeclaredRoster::with_role_migration, restamps the member on its own session;
an undeclared role move holds the member until an apply declares the move or
restores the role. A change of display name, labels, context or instructions
alone keeps the seat. The full set of verdicts is in
The declared apply.
Retiring a declared identity that stays listed does not take it out of the
roster: its member is retired, and the next apply seats it again on its own
session. To take an identity out of the roster (unlist it), apply a complete
roster without it, or unlist that identity alone
(MobHandle::unlist_declared_identity): Core retires its member and ends
the entry. An identity unlisted and listed again comes back on the session it
last had, with its history. Removing an identity
(MobHandle::remove_identity) is different and permanent: it tombstones the
name, which is never an identity or a member again.
Wire Peers
Wiring controls which members can see and message each other.- JSON-RPC
- Python
- TypeScript
wiring rules wire members as they are seated:
auto_wire_orchestrator joins each member to the orchestrator’s members, and
each role_wiring rule joins the members of its two profiles. A Rust host can
take one member out of those rules in both directions with
SpawnMemberSpec::with_definition_wiring(DefinitionWiring::Exempt): the
member is seated with no edge the rules declare, whatever its profile, and no
member spawned, seated dormant or rotated after it is wired to it by them.
Explicit wiring (mob/wire, topology operations) still applies to it, and a
respawn restores the explicit edges it held. The exemption is recorded with
the member, so a restart and a respawn keep it; a rotation
(MobHandle::rotate_member) takes the exemption of the spec it seats. A
durable fork of an exempt member (MobHandle::fork_member, an agent’s
fork_off child, a temporary council’s local participant, a live delegation
worker from MobHandle::fork_member_at_turn_boundary) is seated exempt
too, whatever seating its request states. A member placed on a member host
cannot be exempt.
An exempt member is outside the definition’s role-wide flow dispatch as
well: a flow step on its profile dispatches to it only when the run’s role
targets name it (MobHandle::run_flow_with_role_targets), a flow’s
supervisor escalation never picks it, and a role whose only Active members
are exempt gets a provisioned flow member, as a role with none does.
Send Work
Use member send for direct content delivery. Use the work lane when the caller needs a work reference for the unit it submitted:mob/submit_work returns a
work_ref. One submitted unit cannot be cancelled by that reference; see
Cancel work below.
mob/member_send, SDK Member.send, and mob/submit_work with
origin: "external" require the target’s profile to set
external_addressable: true (the default is false). The definitions above
opt in only the lead as the application’s entry point. For tracked work,
choose a member_ref belonging to an opted-in member. This opt-in is not
required for internal flow dispatch or ordinary peer messaging.
- Member send
- Tracked work
member_ref is an opaque handle returned by spawn, member list, member send,
helper spawn, fork, and respawn responses. Application code should pass it back
to work-lane APIs as-is instead of constructing it from mob_id and
agent_identity.
Cancel work
One submitted unit cannot be cancelled. The host listsmob/cancel_work, and
it is always refused. An otherwise valid call is refused with -32602:
work_ref that is not a UUID is refused one
step earlier, also with -32602 (work_ref must be a valid UUID: ...).
mob/cancel_all_work is a different call. It takes a member_ref and
cancels the work that member had admitted when the cancel was taken, not one
unit: the member’s current run is cancelled at the run’s next boundary, and
the input it held queued is abandoned. Input admitted after the cancel is
untouched. Cancelling an idle member is not an error. On success the result
is {"mob_id": "...", "ok": true}. The cancel is of that member’s work
only: a flow run on another member stays active, and from 0.9.15 a stop of
the mob is still refused while it is (see
Stopping a mob with an active flow run).
Flows
Flows are declarative mob workflows. They let a host dispatch repeatable work without hard-coding all member turns in application code. The classic flow shape is a flat DAG: steps declare roles, messages, dependencies, fan-out/fan-in behavior, optional conditions, and tool overlays. Frame-based flows add nestedFlowSpec.root frames and repeat_until loops.
Both are owned by the mob runtime; support modules such as flow-run projection
are not separate public machines.
Flow parameters and message templates
Flow activation parameters are available to text in step messages. Use{{ params.<path> }} for input, {{ steps.<step-id>.output.<path> }} for a
root-frame result, and
{{ loops.<loop-id>.iterations.<n>.steps.<step-id>.<path> }} for an exact
loop iteration. The .output segment on a root step is optional.
null.
Nested output lookups also depend on the producer step’s output_format:
An explicit format always wins over the schema-aware default. A profile’s
output_schema constrains the member’s structured response; it does not
override a flow step’s text mode. Asking a text-mode producer to return JSON
is therefore not enough to make steps.<step-id>.output.<field> work.
Dispatch and collection still determine the enclosing step-result shape. This
example uses one_to_one with collection_policy: { "type": "any" } so
scan exposes the selected member’s output directly instead of a member-keyed
aggregate. Add this flow to a definition containing the lead and analyst
profiles above:
Advisory nodes (failure_policy)
By default every flow node escalates: if it fails, its frame is classified
Failed and the run fails. Declare "failure_policy": "continue" on a step,
a frame node, or a repeat_until node to mark it advisory. An advisory node’s
failure is still recorded - the node stays Failed in frame state and the step
is still reported failed - but it no longer decides the frame’s terminal
classification.
continue does not keep that node’s dependents running. Dependency
resolution is unchanged by failure_policy. With depends_on_mode: "all", a
single dependency ending Failed, Skipped or Canceled sets the dependent
Skipped; with depends_on_mode: "any", the dependent is set Skipped only
when every dependency ends that way. Either way, a step whose only dependency is
an advisory node does not run when that node fails: the run can still reach
completed, but that dependent’s step status is skipped and a step_skipped
event is emitted for it. If you want work to happen after an advisory failure,
the dependent must have a dependency that can still complete.
That makes continue useful for one shape: an advisory node and a fallback node
feeding a join with depends_on_mode: "any". The advisory node’s failure is
tolerated, the fallback completes, and the join runs on the fallback’s output.
advisory-analyst fails here, baseline-analyst still completes, so
write-up is Ready rather than Skipped and the run completes with the
report written. Drop baseline-analyst and write-up is skipped instead.
Omitting failure_policy means escalate. Frame classification is owned by
MobMachine, so a tolerated failure never becomes a fabricated success: the
failed node remains visible in the frame’s typed state, the step is recorded
failed with a failure-ledger entry, and a step_failed event is emitted. Both
the ledger entry and the event carry the step’s real failure reason. If the
run’s supervisor escalation_threshold is crossed by that failure, the
supervisor is still escalated.
If that escalation cannot be delivered - no member currently holds the
supervisor role, the supervisor’s turn fails, or the escalation turn exceeds
escalation_turn_timeout_ms - the run’s terminal class is unchanged. Failing to
carry out an escalation is not a decision that the run failed, so a tolerated
failure still completes; the undelivered escalation is reported as a
supervisor_escalation_failed event carrying the escalation error, and no
supervisor_escalation event is emitted. The mob is not torn down.
Run and inspect a flow:
- JSON-RPC
- Python
- TypeScript
- CLI
Auditing a run
Every run-result surface carries anaccounting block next to the flow result:
rkat mob run --json, rkat mob attach --json (the detached-run audit path),
the mob/run_result RPC method, and the meerkat_mob_run_result MCP tool. The
two polled surfaces (RPC, MCP) attach it only once the run is terminal, so a
mid-run poll stays cheap and never reports totals for an unfinished run; the CLI
paths wait for terminality before collecting. The block is also absent when
collection failed, which the CLI reports on stderr.
attribution: session_cumulative- each number is the lifetime total of that member’s bridge session. It equals the run’s usage when the session served this run alone; on a mob that has executed several runs it includes the other turns. Nothing in the durable run projection records per-run usage.usage_totalis the sum of the readable members only. Whenmembers_usage_unavailableis non-zero (remote members, archived sessions) it is a floor, and each affected member carries ausage_unavailablereason instead of a zero. Cache and reasoning counters are summed too. Each member’s usage is normalized first, including members whose sessions were saved before 0.8.22, so on every provider cache reads plus writes stay withininput_tokensand reasoning withinoutput_tokens.membersis the mob’s current roster, not this run’s participant set: the run projection records no participation, so a member that took no part in the run is still listed and its session id still rides inmember_session_ids. Exporting the whole pointer set can pull transcripts unrelated to the run.- No monetary cost is reported. The model catalog carries no price data, so
a cost number would be invented;
unpriced_reasonsays so explicitly.
usage, cost, and transcript lines,
the last of which carries the ready-to-run rkat session export-atif <id>
command. rkat run --export-atif writes a trajectory for a single-session run.
Stopping a mob with an active flow run
A mob refuses a stop while one of its flow runs is active.mob/lifecycle
with action: "stop" (MobHandle::stop in embedded Rust) is answered with
the invalid transition error (MobError::InvalidTransition, invalid state transition: Running -> Stopped), the mob stays running and the run goes on.
On JSON-RPC this is error -32602 with the message invalid state transition: Running -> Stopped and no error data.
To stop the mob, wait for the run to end, or cancel it with mob/flow_cancel
(MobHandle::cancel_flow; the run id is the one mob/flow_run or mob/run
returned) and send the stop once mob/flow_status reports the run terminal. A stop
sent right after the run turns terminal can still be refused for a moment;
send it again. A run that mob/flow_cancel does not bring to a terminal
status keeps the stop refused; complete, reset or destroy the mob instead.
Cancelling one member’s work does not end a flow run on another member.
From 0.9.15, after mob/force_cancel, mob/hard_cancel_member or
mob/cancel_all_work on one member, a flow run that is active on another
member is still active and the stop is still refused until that run ends or
is cancelled. In 0.9.14 the stop was admitted and the run went on, no
longer counted (read from the 0.9.14 source, not run on that release).
Observe And Operate
The event log is append-only. For UI and service loops, prefer event cursors or
SDK subscriptions over repeated full snapshots.
Retiring a member also retires every member it spawned with
fork_off,
transitively and deepest first: mob/retire over JSON-RPC, meerkat_mob_retire
over MCP, the browser runtime’s mob_retire, and MobHandle::retire in
embedded Rust all cascade (MobHandle::retire_with_descendants is an alias).
Each retirement runs through the ordinary retire path and every one is
attempted; the first failure is reported, and a retry converges. Members
spawned by other means are not affected.
The mobs the retired member created itself and its delegates end with its
session, at the next start of the host.
Retiring a declared identity that its host’s roster still lists does not
remove it: the identity entry outlives the retirement, and the host’s next
declared apply seats it again. Unlist the identity first (see
Declared identities).
rkat mob force-cancel <MOB_ID> <AGENT_IDENTITY> (RPC mob/force_cancel)
cooperatively cancels a member’s in-flight turn without retiring it - the
member stays in the roster and can take new turns. As the operator remedy for
a wedged member it is legal whenever the mob is running and idempotent:
cancelling a member whose runtime is no longer live, or one already retiring,
converges as a no-op success rather than an admission error. An identity the
roster has never seen is refused with a typed MemberNotFound. A member whose
runtime registration needs a durability reload is refused with a typed
MemberReloadRequired (-32029) until its registration is reloaded. The
cancel is of that member’s work only: a flow run on another member stays
active, and from 0.9.15 a stop of the mob is still refused while it is (see
Stopping a mob with an active flow run).
Persistence
Persistent mobs use SQLite/WAL-backed storage. In-memory storage is used for tests and WASM/browser-embedded paths. The mob store is realm-scoped in the runtime-backed surfaces, so a process restart can recover mob state, members, events, flow snapshots, and work records.Cold resume and cancellation
Reconstructing a stopped mob does not implicitly start its members. Explicit resume prepares the exact session attachments, commits the lifecycle transition, reconstructs members, and then reconciles readiness, topology, and operation bindings. Member construction and external reconciliation run outside the actor loop; status queries and unrelated members do not wait behind one member’s constructor. The publicrunning phase can precede completion of this sequence. Await the
resume operation’s result rather than treating that phase alone as a readiness
receipt. An observer timeout or dropped caller does not cancel process-owned
construction or prove that it did not execute.
Lifecycle cancellation retains ownership until admitted work settles.
Retirement, respawn, and registration reload wait behind the affected member’s
in-flight resume work. Cleanup failures retain exact retry authority; an
unproven cleanup is not a successful cancellation. Compensation preserves
durable session documents and refuses to remove a successor attachment.
A mob restarted over a persistent store after its process crashed, was killed
or exited without a Shutdown takes its adopted identities’ reconcile leases at
once, so each identity’s admission reopens as soon as it converges. The
process proves that its predecessor ended with an exclusive lock on the mob
store’s .incarnation-custody file, which it holds until it exits; never
delete that file while a process may use the store (rkat storage doctor
lists it as mob-incarnation-custody-lock, and rkat storage prune leaves
it). A restart in the same process, a restart while another process holds the
lock, a lease taken without the lock, and a store without a file wait for the
earlier lease to expire (up to 30 s), as before 0.9.5.
Identity adoption (mob/adopt_member_identity_declaration, REST
POST /mob/{id}/members/{agent_identity}/identity-adoption, or
MobHandle::adopt_member_identity_declaration) needs a mob whose runtime has
a runtime store. Reconciliation realizes an adoption by retiring the attached
member and resuming its exact session from the lifecycle record in the
runtime’s store, and a runtime without a store keeps no such record: every
mob on an EphemeralSessionService runs on one. Since 0.9.5 such a mob
refuses the adoption with IdentityAdoptionRequiresPersistentRuntime (error
code CAPABILITY_UNAVAILABLE: JSON-RPC -32020, REST 501; structured data
kind mob_identity_adoption_requires_persistent_runtime) before anything
changes: no intent is recorded, and the member stays attached and serving.
Build the mob over the persistent session service to adopt its members. Its
runtime has a store whatever the realm backend, so unlike a durable worker an
adopted member does not need sessions that outlive the process: a mob in a
memory realm can adopt its members.
Quarantined mobs and degraded members
A host that restores its persisted mobs when it starts (the CLI,rkat-rpc,
rkat-rest and rkat-mcp, and any Rust host of MobMcpState with a
persistent root) restores each mob store on its own. Since 0.9.9 a store that
does not restore no longer fails the restore of the others: it is
quarantined.
- What quarantined means. The store is kept exactly as found and no actor
runs for the mob. Every other mob restores and serves.
mob/listlists the mob underquarantined, not undermobs(so do the MCPmeerkat_mob_listtool, themob_listtool of the RustMobMcpDispatcher, andMobMcpState::mob_inventory), and every call that addresses its id, a create of that id included, answers themob_quarantinedrefusal. The agent-sidemob_listtool lists only the mobs the calling agent may manage, and only a mob’s actor can answer that, so it never lists a quarantined mob: it fails closed. Its structured data carrieskind: "mob_quarantined",mob_id,class,retryable,content_retained,cause(the failure the last restore attempt met) andcode. From 0.9.15: an agent’s own mob tool that names the id is answered the same kind withmob_id,class,retryableandcontent_retainedonly, nocauseand nocode; the cause is in the host’s list and in the host’s log. Through 0.9.14 an agent’s tool is answered every field. Only a mob root that cannot be created, read or listed still fails the whole restore.
- Re-attempts. One re-attempt of a mob’s restore runs at a time: a call
that meets a re-attempt in flight answers the refusal at once and never
waits for it. Any call that addresses a
store_unavailablemob by id can start the re-attempt, before the call checks what its caller may do with the mob. From 0.9.15: an agent’s own mob tool is the exception (through 0.9.14 it starts the re-attempt like any other call): it starts the re-attempt only when the agent’s authority names the mob (a manage grant, a spawn profile scope or a recorded incarnation under the id) or the agent’s session is the creator the mob’s creation record names; any other agent’s call is answered the refusal and starts nothing. A re-attempt runs to its end even when the call that started it goes away (a client that disconnects), and a call that meets a mob a re-attempt is restoring answers the refusal or the restored mob, never that the mob does not exist. A list never re-attempts, and astore_unreadableorresume_refusedmob is never re-attempted on its own. Every start classifies every store again, so a mob of any class restores at the next start once its store does. - Retry after a repair.
mob/restore_retry(RustMobMcpState::mob_restore_retry, owner console only) re-attempts the restore of a quarantined mob of any class and answersrestored,held(a child mob held until its owner resolves),settled(the store held no mob any more, or a child was torn down with its archived owner) ornot_quarantined. A re-attempt that fails answers themob_quarantinedrefusal again, with the class it met, and the mob stays listed with itsattemptscounted. An unknown id is not found. - Degraded members. A member whose restore at a mob start fails on its
own (see the custody deferral above) degrades that member only: the mob
runs, and the member shows
custody_deferredwith the cause, for examplestore_unavailableorstore_unreadable, until a re-attempt restores it. A Dormant member is never read at a start, so a store fault behind it degrades nothing.
ProvenanceUnavailable, and so is
every create by a session while the roster cannot be read at all (a
store_unreadable store), since that session may be one of its members.
Repair or remove such a store to admit them again. A host
decides what a quarantine means for its own boot; Meerkat Meta, for example,
fails its boot when its own mob is quarantined and lists every other
quarantined mob in its hosted set and gateway status.
Durable workers
Since 0.9.5 a profile can make its members durable workers:SpawnContinuityIntent::DurableIdentity, keyed by its AgentIdentity), so
after a restart a host finds the worker with durable_identity_members (see
the next section) and re-attaches it. A durable worker needs a mob whose
member sessions outlive the process. The session service says so through
keeps_durable_member_sessions, which defaults to
supports_persistent_sessions; a host that runs the persistent session
service over in-memory stores answers it false. On any other mob the spawn
is refused with NonDurableService (error code CAPABILITY_UNAVAILABLE,
structured data kind mob_non_durable_service) before anything is created,
instead of spawning a worker that would not survive a restart. durable lives
on the profile only, with no separate spawn flag, and an explicit continuity
intent on a Rust spawn wins over it. A host older than 0.9.5 refuses a profile
that sets it.
A durable worker restarts like every member of a persisted mob: it is rebuilt
on its session, and a run it was in continues from its last committed round
(see Durable runs). A host that tracks
work outside the session, such as a work item another worker may have claimed
meanwhile, can stop such a run from continuing with a resume gate
(MobBuilder::with_durable_resume_gate); see
Durable runs: resume gate.
Member binding history (Rust hosts)
Since 0.9.4 a Rust host can ask which incarnation of a member held which session, as a store read that never waits behind the mob actor:MobHandle::member_binding_history() (also on read_handle()) and its
single-read forms member_binding_for_session, last_member_binding,
durable_identity_members and identity_management. For a stopped mob,
MobStorage::read_member_binding_history(path) reads the same history from
the persistent store, read-only. See
Member binding history
for what each read answers.
Operator instructions
A host gives a mob’s members standing System-plane instructions by slot (a namespace the host owns and a key), revision and body, for a target: every member ({"kind": "mob"}, OperatorInstructionTarget::Mob in Rust),
the members of one profile ({"kind": "profile", "profile": ...}) or one
member ({"kind": "member", "agent_identity": ...}). A target
is a predicate, not a reference: a member added later, or moved to the
profile later, receives the record then, and a target that names a missing
profile or member matches nobody (the receipt’s matched_members shows
zero). Since 0.9.11 the verbs are on JSON-RPC (mob/set_instruction,
mob/revoke_instruction and mob/instruction_status; see
RPC operator instructions) and in the Python
and TypeScript SDKs. The Web SDK has none: browser sessions keep no durable
session document, so every delivery would be refused
(durability_unavailable).
MobBuilder::with_member_instruction_delivery_concurrency); deliveries
through live sessions run in their own lane. An exact repeat answers as a
duplicate; a lower revision, the same revision with other content, a body
over 256 KiB, a 65th record or more than 1 MiB of Active bodies in one
member’s snapshot, and a
completed or destroyed mob are refused typed
(OperatorInstructionError::code; on the wire error.data.instruction_set_code).
A member’s snapshot is the shared records (targeted at every member or at a
profile) plus the member-targeted records that name the member (and, for a
fork, the records written for the first member of its fork family and its
forks, see include_forks below). Any other
member-targeted record is not in it, so a member’s session and its
status row do not grow with the number of other members that hold a record.
The two limits bound that snapshot, the shared records and the member’s own
together: at most 64 records (slot_limit_exceeded) and 1 MiB of Active
bodies (active_set_budget_exceeded). A mob can therefore hold a
member-targeted record for each of any number of members, such as one
standing text per agent. Shared records are in every snapshot, so they stay
within 64 records and 1 MiB as before, and a mob that already holds 64 shared
records has no room for a member’s own. A higher revision that would move a
record into a full snapshot (a member’s record retargeted to every member,
or to a member whose snapshot is full) is refused the same way.
A revoked slot keeps its record (its
effect Absent), so it counts toward the 64 records of the snapshots it is in
(every member’s for a shared record, its one member’s for a member-targeted
record) until a reset clears the
records. A record that names a member the mob no longer has is in no
snapshot, but the limits still count it under the name it targets: they
count a member-targeted record by its target, seated or not, so the records
of a retired or never-seated name keep taking room from new shared records
(a shared record must fit beside the most records any one name holds, and
beside what another name’s records pass to its forks, see below) until
they are written with another target or a reset clears them. A refused
shared write does not name the member whose snapshot is full; the status
read’s desired list shows each record’s target. Writes need the AdminHost scope and are never an agent tool. On a
governed runtime (a runtime with a native work authorization host) every
write and revoke is refused prepared_authorization_required before anything
is journaled: such a write needs prepared authorization, which this release
does not mint, so a retry is refused the same way.
A member-targeted record covers the member’s forks when it is written with
include_forks
({"kind": "member", "agent_identity": ..., "include_forks": true},
OperatorInstructionTarget::Member { include_forks: true, .. } in Rust).
Without it, the default, the record is its member’s alone and a fork of the
member runs without it, as before. With it the record follows the member’s
fork family from its root. Written for a member that is not itself a fork,
it selects that member and every fork below it, forks of forks included.
Written for a member that is itself a fork, it selects that member only:
that fork’s own forks
follow the records written for the first member of their family, not for
the fork between. Each member’s row of mob/instruction_status lists the
records of its snapshot, a fork’s included, so the rows show which members
a record selects, and the
write’s matched_members counts the seated ones. A durable
fork (fork_off, MobHandle::fork_member and its bounded and detached
forms, mob/fork_member, a
live delegation’s fork) has its snapshot recorded in its committed session
before it is seated, and a helper fork
(a spawn with a fork launch mode) is born with it. A seated fork then
follows the record like any member it selects: a higher revision and the
revoke reach it as entries appended at its next fresh run start, and nothing
already in the fork’s conversation is rewritten. A fork taken after the
record was revoked gets the record’s slot with no body: nothing of it is in
effect in that fork. A higher revision that writes the record without
include_forks (or for another member) takes it from the seated forks: each
drops it at its next fresh run start. A durable fork copies its
source’s conversation up to the fork point, so its model also reads the
source’s earlier entry
rows, each as a header without a body
([meerkat-inherited-operator-instructions-v1],
status=not_active_in_child): those are history, and the fork’s own entry
after them is what applies. The same recording puts the shared records
(targeted at every member or at its profile) in every durable fork’s
snapshot before its seat; before, a fork that started its own turn could
run it without them. A
fork whose snapshot cannot be recorded is not seated
(fork_member_provision_failed): once the mob has recorded its first
operator instruction, a durable fork fails over a child session that refuses
the record for any reason, where a seated child was only blocked before
(another owner of the session’s target, or a session service that keeps no
durable session document or does not forward the record). The mob holds each
fork’s lineage (the
first member of the family it descends from) with its roster: it survives a
restart, it follows the fork’s identity through a respawn, a rotation and a
host’s adoption, and it ends with the fork’s
identity. The lineage names that member as a target does, by name, across
lives: the forks of a member that was removed keep following the records
written for that name until those are revoked, and a record written with
include_forks for a new member seated under that name, as a fork or not,
reaches the earlier
member’s forks too (the one exception to “that member only”). A record for
a member and its forks counts in the
snapshot of every other name as well (any name may be seated as a fork of
that member), so such a write is refused slot_limit_exceeded or
active_set_budget_exceeded when it would not fit beside the most another
name holds: a member that holds 64 records of its own blocks every
include_forks write for another member, and one name’s records for forks
lower
the room every other name has for its own. The refusal does not name that
member: the mob logs a warning at the refused write that names the member
whose snapshot would be full and what fills it (the shared records, its
own, and what another member’s records pass to forks; with no
member-targeted record it says that the shared records alone fill every
snapshot), and the status
read’s desired list shows each record’s target. No fork is refused
at its seat for room. The participants of a
temporary council are seated in the council’s own mob, which holds none of
their source’s records: they are not covered. Only a fork the mob itself
takes (the routes above) joins a fork family: a session forked outside the
mob and seated later as a member is not a fork for this rule, and neither
is a fork seated by a release before 0.9.15 (see
Upgrading).
A placed helper fork whose seat the mob repairs after its host stopped in
the middle of it is seated outside its fork family (the mob logs a warning
that names the member).
mob/instruction_status (MobHandle::operator_instruction_status, scope
ReadHistory) reads the
records and, per member, its residency, its delivery state (current,
owed with the last retryable failure, or blocked with a reason), and per
slot of its snapshot what the member should have, what its session recorded
and the entry
that put it in effect, with the reason a change is still pending (the
session’s next run start, a model that lowers System rows only in the
leading prefix, or one that cannot carry them at all). It reads committed
session state and never wakes a member; a member whose session cannot be
read is reported with the reason, and the page still answers. A member is
blocked for one of owned_by_session, owned_by_other_mob,
slot_managed_by_activation, durability_unavailable (its session service
keeps no durable session document) or member_host_unsupported. The last
failure is diagnostic and is not kept across a restart. The reasons
and the residency are advisory and may grow: the SDKs keep a value they do
not know as {"unknown": <the value>}. A member’s session also publishes an
operator_instructions_applied event for each entry it applies (an
observation; the status read is the authority).
A member’s session target belongs to its mob: another mob’s delivery to it
is refused (owned_by_other_mob) until the owning mob releases it, which a
mob does when a member whose session its host keeps retires and when the mob
is destroyed. A destroy goes on past every member whose release does not land
and answers incomplete naming each of them, still seated, so one retry
releases them all; a member session whose operator-instruction state does not
read has nothing to release and never holds up a retirement or a destroy. A
member’s session refuses its own host’s session-level writes
(session/set_instruction and
SessionServiceInstructionExt::set_session_instruction answer
owned_by_mob) from the moment a mob seats it, even before the mob writes
anything, so the mob’s first write never replaces a write the host was told
was recorded; write to a member through mob/set_instruction
(MobHandle::set_operator_instruction) with a member target instead. A
dormant resume seat stamps the member binding
at the seat, without a wake, on a session that names none. The member binding
a mob stamps on a session is never cleared, so a session a mob seated keeps
refusing its host’s own writes after the member retires with the session kept,
after a failed attach rolls back, or after a dormant seat that stamped it is
refused. A failed attach, or a member that retires before the mob’s first
write, therefore leaves the instructions its host wrote before the attach in
effect, and its host can no longer change or revoke them: revoke them before
attaching a session that may leave the mob. A host’s
standalone adoption (Session::rebind_instruction_owner to the session) of a
session a mob built or seated is refused owned_by_mob too. A member seated
over an existing session whose own host wrote its target before (an attach)
hands that target to the mob at the mob’s first delivery: the mob’s snapshot
replaces what the host wrote. That first delivery waits until the member’s
spawn has activated, so an attach whose activation fails rolls back with the
host’s session untouched. An attached member’s first activation turn (its
kickoff or initial message) therefore runs before the mob’s instructions
arrive; they apply from its next fresh run start. A blocked member stays
blocked until a write with a new revision of a slot that selects it, a new
seat of the member, or a restart; an exact repeat of a write does not unblock
it. A reset clears the records and keeps the generation moving forward, so a
session never sees an older snapshot replace a newer one. A placed member
(one on another host) receives its snapshot on its member host: the mob
sends it over the supervisor bridge, and the host records it in the
member’s session the same way, without waking the member. A member placed
by a spawn that mints its session on a 0.9.11 member host is born with its
snapshot too: the mob asks the host’s features once for the spawn, the
host records the snapshot in the session it mints, and the member’s first
turn (its initial message or kickoff) already has the instructions. On an
older or a governed member host the member is born without them: its
first turn runs without the instructions, and its delivery then blocks it
member_host_unsupported, as follows. In a mob whose
hosts run different releases, a placed member whose member host runs a
release before 0.9.11 is reported blocked (member_host_unsupported): its
host refuses the record before any effect, nothing is recorded, and the
member stays blocked until its host runs 0.9.11 and a write selects it
again (or it is seated again). A placed member on a governed member host
(its runtime has a native work authorization host) is blocked
member_host_unsupported too: the host refuses the record
prepared_authorization_required before anything is written, since this
release provides no prepared authorization for instruction writes. A
snapshot whose record would not fit the supervisor bridge’s frame (about
1 MiB with its envelope, which a snapshot’s 1 MiB active budget can reach) is
not sent: the placed member is blocked member_host_unsupported until a
write changes its snapshot. The status read reports a placed member’s
delivery but does not read its session on the member host
(effective_read is unreadable). A member’s status row lists a record
that moved to another member until the member’s session has revoked it;
for a member whose session the read does not reach (a placed member, an
unreadable session) such a record leaves the row at once, and the member’s
delivery reads owed until its new snapshot is recorded. A mob that recorded operator
instructions journals
them at mob event schema 10, which a release before operator instructions
refuses for every mob of that realm, not only that one: see
Downgrading to 0.9.8 before any
downgrade. rkat storage doctor lists such mobs
(operator-instruction-mob-events); it reads the mob databases on disk
(mobs/<name>.db in each realm’s state directory), not a host store.
Shutdown, destroy and admission (Rust hosts)
Every exportedMobHandle clone shares one owner token. A mob whose admission
is closed refuses new work with MobError::AdmissionClosed and a
MobAdmissionCloseReason, while stop, shutdown, destroy, retirement,
cancellation and reads keep working. Since 0.9.4 no mob actor exit leaves
admission open:
An earlier close (
IdleClose, LastOwnerReleased, StandDown or one of the
above) keeps its reason. After any of them acquire_owner() is refused and
admission_closed() reports the reason; 0.9.3 and earlier could still hand out
an owner lease on a shut-down, destroyed or lost actor.
When a mob start’s durable recovery fails before the actor admits a command
(DurabilityFailStop), MobHandle::status and MobHandle::resume return the
failure, typed, as MobError::SharedLifecycleFailure. For a member’s session
it is MobError::MemberStartupRecoveryFailed, which names the member, its
session and the cause, for example a store error while the start restores a
retiring member’s cleanup delivery. Other commands fail with
ActorCommandChannelClosed. An actor that fail-stops later, on a durable
mutation whose outcome stayed uncertain, reports MobError::DurabilityFailStop
the same way: its cause is that mutation’s own error, the first one when
several stayed uncertain, and the command that hit it got the same error,
shared in SharedLifecycleFailure. The record classifies as the admission
close it stands for (no wire code, failure class MobRejected, mob/list
cause internal) and carries no durable resume hold, even when its cause
does: the mob needs a restart, not a parked member. Before 0.9.5, status
reported only “mob actor reply channel closed”, and the cause reached only the
host’s error log.
MobHandle::lifecycle_observation() (and the same method on MobReadHandle)
reads that answer without asking the actor: the last phase the actor
published and, once the actor has stopped, failure. That is the cause
status reports or, for an actor task that ended while the mob was live and
recorded no cause, the closed-actor error. status_observation_snapshot() is
the phase alone and keeps a stopped actor’s last live phase. mob/list, the
MCP meerkat_mob_list tool and the SDKs’ mob listings give such a mob’s row a
typed failure (cause, message, code, structured_data): present
means the mob’s actor has stopped; status is the last machine phase. Before
0.9.5 the row read Running.
From 0.9.15 the observation also carries owed_intent: the Stop, Complete,
Reset, retirement of every member or Destroy that was durably begun and has
not landed, or None when the mob’s machine state owes nothing. A resume waits
only a short bounded time for the first attempt at a recovered Stop or
Complete, so the phase read right after it can still be Running while the
Stop is owed. A host that acts on the phase after a resume (lifting a Stopped
mob, for example) first calls
MobHandle::wait_until_no_lifecycle_intent_is_owed(), which returns the phase
once nothing is owed and has no bound of its own. Two reads say “nothing
owed” while the mob’s log still owes an intent: the read handle inside a
pre-activation hook of a resume (the open intent is applied after the hook;
read it from the handle the resume returns), and the handles of an actor that
a graceful Shutdown ended over an owed Stop (they read Stopped; the next
resume reports the Stop as owed and drives it). When the actor ends before the
intent lands, the wait returns a typed error: the fail-stop cause, the
closed-actor error for a task that ended in a live phase, or
ActorCommandChannelClosed for an actor that ended in a terminal phase with
the intent still owed; after a graceful Shutdown over an owed Stop the wait
returns Stopped, as those handles read. That Stopped is the answer of
that actor: a host that records “stopped” durably takes it from the next
start’s wait, not from a wait a Shutdown may have ended.
A member whose cleanup delivery the start cannot restore (a retiring member,
or any member bound to a session of a Stopped or Completed mob) does not fail
the start when the failure is the member’s own and came before the restore had
an effect: another runtime owner on the realm hosts its session, its hosting
claim cannot be taken, its durable tail could not be proven, or a read of its
session state failed before any settle of its durable tail ran
(store_unavailable when the store could not serve now, store_unreadable
when the state cannot be read or decoded). The start defers
that member’s custody restore: the mob runs, the member keeps its state and its
session untouched, and MobHandle::member_status shows the deferral as
custody_deferred (the restore and the cause its last attempt met; both
vocabularies may grow, and the SDKs report a value they do not know as unknown
instead of failing the read). The member is degraded, not Broken. A retire,
redrive_retirement or respawn of the member re-attempts the restore first
and answers MobError::MemberCustodyDeferred while it still cannot restore;
an explicit resume re-attempts it once and still succeeds. A restore
that owes work (a retiring member’s cleanup) is also re-attempted on its own,
with backoff: once it succeeds, the retirement completes. A degraded member of
a Stopped or Completed mob is re-attempted only by an operation on it, an
explicit resume or the next start; no timer reads its state in the background.
A destroy of the mob re-attempts every deferred restore first and is refused
with MobError::MemberCustodyDeferred while one still cannot restore, so a
member whose stored state cannot be read (store_unreadable) holds the
destroy until that state is repaired. A divergence of the mob’s own records, a
failed settle of the member’s durable tail (a write), or a failure after the
member’s registration was prepared, still stops the start.
While a Shutdown drains, a command the close refuses (spawn, respawn, work, a
flow run, wiring, a host bind, a live commit, resume, reset or reload) fails
with AdmissionClosed, and every other request the actor answers (stop,
destroy, complete, retirement, reads, MobHandle::status) fails with
MobError::ShutdownInProgress: retryable, authority retained, wire code
SESSION_BUSY. MobHandle::stop returns it at once. Retry once the Shutdown
has finished, after which requests get ActorCommandChannelClosed (the actor
has exited or is exiting), or has failed: a failed Shutdown keeps the actor
serving with admission closed for its life, and Stop, Shutdown and Destroy stay
available. 0.9.3 and earlier answered all of these with
ActorCommandChannelClosed. A command sent while the actor exits after a
Shutdown now fails instead of waiting forever.
shutdown_with_report(ShutdownOptions) returns a MobShutdownReport with one
MemberShutdownOutcome per member; is_clean() is true when every member was
unregistered. Since 0.9.4 a member reported UnregisterPending { stage, settlement, .. } carries a MemberUnregisterSettlement that keeps following
the member’s exact runtime registration after Shutdown returns, also after the
mob’s actor exited: try_state() reads Settled, Pending, Failed or
HandedOver without waiting, and settle_within(deadline) waits, admitting
the unregister itself when Shutdown’s deadline passed first. HandedOver
means a later owner (a resumed mob, or any caller that reused or attached to
the registration) took it over; the settlement then touches nothing. A member
whose registration is durability-degraded is now unregistered after the
runtime’s reload-required discard (reported at stage
durability_reload_discard while the runtime refuses the unregister until
then); 0.9.3 and earlier left it registered until the process exited. Stop and
Shutdown report such a member’s run and run starts as ReloadRequired instead
of failing on it, and a Shutdown holds every member’s run starts, reporting the
first failure in roster order.
Since 0.9.5 a graceful Shutdown keeps a durable member’s work for the mob’s
next start. MobHandle::shutdown() and ShutdownOptions::default() take the
default QueuedInputsOnShutdown::RequeueDurable, which resolves per member:
QueuedInputsOnShutdown::Requeue(RequeueMaxAge::DEFAULT), a 15 minute max
age, for a member whose session commits every round (the mob’s session
service keeps persistent sessions and its runtime has a persistence store),
and QueuedInputsOnShutdown::Stop, the 0.9.4 behaviour, for any other
member. A member already retiring is stopped either way. To choose for every
member alike, pass
ShutdownOptions::default().with_queued_inputs(QueuedInputsOnShutdown::Requeue(max_age)),
where max_age is a RequeueMaxAge (RequeueMaxAge::new(duration) refuses
a bound under one millisecond, zero included, or over 24 hours with
ShutdownOptionsError), or .with_queued_inputs(QueuedInputsOnShutdown::Stop),
which ends every queued input as stopped and cancels running turns, as in
0.9.4.
.with_queued_inputs(QueuedInputsOnShutdown::Keep) keeps what the Shutdown
finds as it is, for a host that boots over a store another host serves,
holds its members’ run starts and admits nothing to them (a candidate
launch). A member whose runtime registration served no live work keeps every
input the registration recovered as the Shutdown finds it, with no requeue
mark, and the run its session left open, and an autonomous member’s kickoff
stays open: the next start recovers and resumes them. A run left open whose
stop was asked after the start registered it is the exception: the inputs
bound to it end as stopped, and no later start resumes it. The boot’s
ordinary recovery, which every start runs, may still normalize a staged
input whose run cannot continue to queued before the Shutdown; the Keep
leaves that as it is. A member whose registration served live work (an
admitted input, a deduplicated resubmission or a recycled registration) is
stopped, as under Stop. While the host holds a member’s run starts, a
kickoff an earlier Shutdown kept in flight is not re-attached at the
member’s restore but at the host’s release_member_run_starts, so a host
that never releases the member serves it no work and its Keep Shutdown
keeps the kickoff open.
Under Requeue the Shutdown holds a member’s run starts and does not cancel
its running turn. The member’s runtime stop suspends a run that commits
every round at its last committed round: the run’s inputs stay bound to it,
with the waiter answer and the max age below, and the run continues from
that round at the next start, before the kept queued inputs run (see
Durable runs: graceful stops and deploys).
An autonomous member keeps its running turn the same way, and its kickoff
stays open: the next start resumes the run under the same objective, and
when that run is the kickoff’s, the kickoff resolves when it ends. A
retiring or host-placed autonomous member is still interrupted, with its
kickoff closed. A draining deploy that first stops the mob with
MobHandle::stop_with_options(StopOptions::default().leave_runs_running())
and drains with MobHandle::drain_stopped_runs leaves such a member’s
kickoff open as well, so the Shutdown that follows decides: its default
keeps the kickoff and the run for the next start, and
with_queued_inputs(QueuedInputsOnShutdown::Stop) closes the kickoff and
cancels the run. If no Shutdown follows, the kickoff stays open and resolves
when its run ends. A plain Stop, which interrupts runs, still closes the
kickoff. The runtime stop still cancels a run that commits no rounds,
such as a live bridge run: that turn’s own input ends cancelled, and its
user message is not in what a kept input’s turn sees.
A member input still queued behind the running turn is kept when it is a
durable prompt in the queue lane that is neither directed nor bound to an
objective, as a host human input submitted with HandlingMode::Queue is. It
stays in the member’s input store, its waiter ends with
BoundedTurnFailure::RequeuedAcrossStop, and the mob’s next start runs it
once, in admission order, before new work; a same-key replay after the
restart re-attaches to its result. Do not resubmit it. If more than
max_age has passed since the Shutdown kept it when a start finds it still
queued (every such start checks again), it ends as abandoned
(requeue_expired) without running, and a same-key replay answers
BoundedTurnFailure::RequeueExpired: resubmit it if it is still wanted. The
age counts from the Shutdown, not from the input’s admission: max_age
bounds how long the member may stay down, however long the input waited
before the Shutdown.
Steer, ephemeral, event, directed and objective-bound inputs that are not
bound to the running turn’s run, and peer messages, end as stopped. A
member’s kickoff is bound to its objective, so a kickoff still queued at a
requeue Shutdown ends as stopped and is recorded failed, as with Stop.
Keeping queued inputs is guaranteed only for a member whose run starts the
Shutdown holds (MemberRunStarts::Held): a member that cannot hold them may
start a queued input once its running turn is cancelled, and that input then
belongs to the cancelled run. 0.9.4 cannot open a store that holds a kept
input still queued or a kept run, and cannot read an expired input’s row, or
the row of a kept queued prompt that 0.9.5 ran, after that (see
Upgrading): shut a mob down with
QueuedInputsOnShutdown::Stop before you downgrade it.
A mob restored with an unfinished destroy starts with admission closed before
MobBuilder::resume returns its handle, so nothing is spawned into it while the
destroy is retried. A recorded placed lifecycle intent, such as a Reset, is not
retried while admission is closed; it stays recorded and the next start drives
it. MobError is #[non_exhaustive] since 0.9.4: an exhaustive match needs a
wildcard arm.
Since 0.9.4 a destroy whose member’s boundary commit fails while the destroy
waits on that member’s ingress detach reloads the member from its durable
store, retries its retirement once, and names it in
MobDestroyReport::reloaded_members; 0.9.3 answered Incomplete.
A destroy also reloads, before it starts, every member whose runtime requires
a durability reload. When the runtime refuses such a reload as repair-blocked,
for example while a boundary commit past its deadline has not reported its
durable outcome, the destroy answers Incomplete. Since 0.9.5 its report also
lists the member in MobDestroyReport::member_reload_refusals
({"member": ..., "refusal": {"kind": "repair_blocked", "reason": ...}}
in the destroy_report of the mob_destroy_incomplete data): retry the
destroy later. The SDKs read that report typed, as a WireMobDestroyReport:
mob_destroy_incomplete_report(error) in Python and
mobDestroyIncompleteReport(error) in TypeScript and @rkat/web. A refusal
in the reload pass before the destroy starts leaves the mob unchanged. A
refusal later in the destroy comes after its cleanup has begun: the mob’s
admission is closed, and some members may already be retired. A member that
degrades only after those reloads, and whose reload the runtime refuses while
the destroy archives its session, is listed the same way; its errors entry
then carries the text of MobError::MemberReloadRepairBlocked. Any other
reload failure stays text in errors only.
A retire reloads such a member as well, at its quiesce stage. When the runtime
refuses that reload as repair-blocked, the retirement sticks: retire answers
MobError::MemberRetirementStuck, and since 0.9.5 its cause is
MobError::MemberReloadRepairBlocked (session_id, reason,
evidence_digest; structured data kind mob_member_reload_repair_blocked
with retryable: true, wire code SessionBusy) instead of a session error
with the refusal as text. Redrive the retirement with
MobHandle::redrive_retirement once the boundary commit has reported its
outcome; the redrive reloads the member and completes the retirement.
A retire that started before the member degraded waits at its quiesce stage
on the boundary commit instead, and reaches its ingress detach once the
commit reports, with the member still degraded. Since 0.9.5 the ingress
detach reloads the member as the quiesce does and retires the cold successor,
so the retirement completes: retry a retire that answered
MemberRetirementInProgress. When the runtime refuses that reload as
repair-blocked, or the member degrades just after the stage’s reload check,
the retirement sticks with the same MemberReloadRepairBlocked cause; redrive
it. Since 0.9.5 the stuck error’s own data (kind mob_retirement_stuck)
carries the cause’s data as cause_data, so a client routes on the cause’s
kind rather than on its text.
Admission in custom Rust session services
A customMobSessionService that executes standalone turns must implement
start_turn_with_admission_notification. Send its notification only after
generated admission and command handoff; spawning a task or waiting for
terminal completion is not a substitute. The built-in ephemeral service
provides this contract. The default custom-service implementation returns
Unsupported, rather than allowing a later delivery or retirement to overtake
an unacknowledged admission.
Supervisor Rotation
mob/rotate_supervisor is a synchronous-looking view over a durable
operation. The mob records a stable operation ID and the complete target
authority before sending the one-way handoff to any member. Each member fences
the old epoch, advances the handoff independently, and exposes a durable
pending, completed, or rejected receipt for that operation ID.
A caller timeout means only that the terminal receipt was not observed before
the deadline. It does not cancel or roll back the member operation; a retry
uses the persisted operation ID and resumes observation. See
Delivery, Interaction, and Durable Operations
for the protocol boundary and recovery semantics.
Multi-Host Placement
Use a member-host daemon when the controlling mob should place and supervise members on another machine. The daemon uses its own restart-stable realm, ideally a dedicated realm or context root; realms are not distributed and do not need to match the controller’s realm.--isolated is rejected because it
would select a new throwaway realm on every restart and lose the daemon’s
durable member-host state.
advertise_tcp must be dialable from every member host. Without an explicit
controller endpoint, mixed-host route installation remains pending and fails
closed rather than publishing a process-local address.
The descriptor handoff above does not require network pairing. If you enable
the host’s pairing branch, provide its runtime-only secret through
--pairing-password-env <ENV> or --pairing-password-file <PATH> so it stays
out of process arguments. These options conflict with each other and with the
compatibility-only --pairing-password <PASSWORD> form. The secret is never
written to [mob_host] configuration and must be at least 32 bytes.
The bind report’s host id is the placement value for mob/spawn and
mob/spawn_many. Omitting placement keeps the member local:
mob/wire with an
external peer descriptor). The edge is the same controller-owned external
edge as for a local member; its trust row is installed on the member’s host,
tracked as a route-install obligation (mob/route_installs reports it under
outstanding_external until the host acknowledges it), and reinstalled after
a host restart, a member revival, or a controller restart. Unwiring removes
the host row before the edge is committed as unwired. Read a placed member’s
dialable endpoint (address and key) with MobHandle::member_endpoint_status
or member_peer_endpoint.
Security note: wiring a placed member to an external peer makes that member’s
runtime on the member host trust the descriptor and dial its address. Only a
caller with wiring authority over the mob can do this (the same admission as
wiring a local member), but treat external descriptors as trusted input: the
member host will connect to whatever address the controller installs. A wire
is refused when the member’s host is not bound or its carrier binding is
inactive, and when the member is already wired to the same peer id under a
different name.
The placed session id is diagnostic identity in the owning member-host realm,
not a controller-local session/* handle. Drive the member by
agent_identity, and read its transcript with mob/member_history. Generic
session history, transcript revision, and transcript edit operations are not
proxied across hosts in v1.
Placed-member operator authority
A placed member’s mob operator tools (spawn_member, list_members,
retire_member and the rest) are upcalls to the controlling host, which
admits each one against the operator authority it recorded when it compiled
the member. A session build hook on the member host does not change that
authority.
A placed member whose profile enables mob tools and that has no ceiling keeps
manage scope over its mob, including the privileged spawn arguments: it can
spawn any profile of the mob, with a runtime mode, a launch mode or a
requested tool access policy, and it can list and retire every member. That
is more than a local member gets by default, which holds no manage scope.
Controllers narrow a placed member by setting a ceiling on its spawn request.
In a Rust host, set it with SpawnMemberSpec::with_operator_authority_ceiling,
or from a SpawnMemberCustomizer with set_operator_authority_ceiling:
without_manage()drops manage scope. The member then spawns only the profiles it holds, never with a privileged spawn argument, and it is refused the operator tools that need manage scope, such aslist_membersandretire_member.with_spawn_profiles(...)keeps only the listed profiles (a profile the mob does not define is ignored). It needswithout_manage(), because manage scope admits every profile: a ceiling with both is refused withMobError::OperatorAuthorityCeilingRefused(SpawnProfilesWithManage).without_create()andwithout_mutate()drop create and profile-mutation authority.
respawn or
respawn_with_successor_spec) keeps it: a spawn customizer sees the recorded
ceiling on the respawn request, and a ceiling set on that request replaces it.
When the record cannot be read, the respawn is refused before anything is
retired.
A ceiling also bounds every member that member spawns itself (its
spawn_member and spawn_many_members calls): the child is compiled with at
most the authority recorded for the member that asked for it. A ceiling on the
child’s own spawn request, or one a spawn customizer sets, can narrow that
bound but never widen it, so a narrowed member cannot regain authority through
a member it spawns, and the bound holds across the child’s restarts and
respawns. A member without a ceiling bounds its children by nothing more than
the default. A flow run needs manage scope. When a member that has it but
carries a ceiling starts a flow that would provision a member, the run is
refused before it starts: flow provisions are local, and no ceiling can bound
a local member. Its flows whose roles already have members still run.
A ceiling on a spawn without a placement is refused (UnplacedMember), since
a local member’s operator authority is set by its own session build. The
JSON-RPC, REST and SDK spawn surfaces do not carry a ceiling.
Placed-member portability
Remote placement compiles one digest-coveredPortableMemberSpec. Admission
rejects host-local or secret-bearing behavior before any materialization
request; it never silently strips the field and launches a weaker member.
The typed non-portable causes cover:
- Rust tool bundles (
rust_bundles, and since 0.9.4 bundles a profile opts into withtools.bundles), per-spawn external tools, mob-default external tools, and a default LLM client override - an in-process compaction curator override
- a host-surface MCP allow-list, an inherited tool filter, and WorkGraph tools
- non-empty shell environment, MCP stdio environment, or MCP HTTP headers
backend or
RuntimeBinding: one member has one execution transport story.
WorkGraph commitments and stores stay on the controller, so a profile that
requests WorkGraph tools cannot be placed remotely. Schedule tools are
portable. Memory and declarative MCP servers can be portable, but only when
the bound host advertises the required capability and the MCP declaration does
not carry secret environment or header values. The same admission ladder
fails closed when the host lacks required autonomous-member, durable-session,
tracked-cancel, protocol-v4, memory-store, or MCP capability.
Operator surface boundaries are intentional:
The identity-routed mob member live family leaves the transport to the
member’s own live host, for local and placed members alike. A
transport = "webrtc" request to mob/member_live_open returns a WebRTC
bootstrap answered with live/webrtc/answer on the same JSON-RPC server, and
a host that serves no WebRTC refuses it with LiveTransportUnsupported. A
member host started by rkat mob host serves WebSocket only, so a placed
member’s channel is WebSocket; there is no controller session/* path to a
placed session.
Use mob/hosts, mob/route_installs, and mob/member_history (or the
matching CLI commands) for placement diagnostics. Python and TypeScript expose
the full typed RPC family. REST and public MCP intentionally expose the three
read-only observations, while host binding, grants, hard cancel, and member
live control remain outside REST and public MCP. Host binding, grants, and
member live control also have explicit CLI verbs. On public transport
surfaces, hard cancel is RPC/SDK-only; trusted in-process MobHandle callers
also expose it as hard_cancel_member (mob/hard_cancel_member, Python
Mob.hard_cancel, and TypeScript Mob.hardCancel). rkat mob force-cancel
is the distinct cooperative boundary cancel.
Browser/WASM mobs remain single-host and reject non-local placement through
their typed capability boundary.
Embedded and explicitly delegated non-owner console principals need control
scopes. Stock v1 RPC, REST, stdio, MCP, and CLI entrypoints mint the owning
console principal; manage narrower delegated principals with rkat mob grant,
rkat mob revoke-grant, and rkat mob grants. Scope denials and
host/cursor/fence failures are typed consistently across console surfaces.
External Members
Most members are normal session-backed agents. External members are advanced: they require an external runtime binding with a concrete address and trusted peer identity so the orchestrator can route supervisor bridge traffic to the right process. This binds one already-running external member; for a managed host that can materialize multiple placed members, use the member-host flow above. For a remoterkat process, start it with the signed comms listener and
write the binding file:
--comms-binding-out writes the current external binding shape: kind: "external", advertised address, Ed25519 public identity, and typed
bootstrap_token. Current mob supervisors require that typed bootstrap token
for external bridge binding; a bare External backend tag, a raw address, or a
query-string-only bootstrap token is rejected.
Use external members only when the member must run outside the local Meerkat
runtime, such as another host, sandbox, or service process.
Live Channels
Live channels execute in the owning session. For a mob member, address the member by identity throughrkat mob live open, RPC
mob/member_live_open, Python Mob.member_live_open, or TypeScript
Mob.memberLiveOpen. That path is placement-transparent: the mob checks the
member, and the member’s own live host serves the requested transport (the
owning host’s WebSocket endpoint, or a WebRTC bootstrap answered with
live/webrtc/answer on the same JSON-RPC server) or refuses it typed.
The generic session-scoped live/open refuses a mob member’s session; the
member’s mob opens its channels. For a gpt-live-1 voice channel on a member
local to the JSON-RPC server, pass execution_identity to
mob/member_live_open with transport: "webrtc"; a placed member refuses
it in 0.9.5. A placed session cannot be opened through the controller’s
generic session surface.
See Live channels.
Mobpacks
A mobpack packages a mob definition and optional assets into a portable artifact:mob web build copies required prebuilt
wasm-pack output into that bundle; it does not compile wasm32.
Troubleshooting
See Also
Mob architecture
Runtime ownership, member identity, flows, persistence, and live-channel boundaries.
Mobs concept
The conceptual model behind members, profiles, wiring, and host-vs-agent surfaces.
Mobpack
Package, sign, validate, deploy, and build browser-target mob artifacts.
