Unified Runtime

This page documents MobKit v0.8.34 (mirrored from v0.8.34). The unified runtime is the production entry point for MobKit. It bootstraps a Meerkat mob and MobKit’s module subsystems together, merges their event streams, and manages the combined lifecycle through a single handle.

Builder paths

For new hosts, build from a mob definition. definition_path loads TOML, while definition accepts an already parsed MobDefinition:

use meerkat_mobkit::UnifiedRuntime;

let runtime = UnifiedRuntime::builder()
    .definition_path("config/mob.toml")
    .persistent_state(".mobkit/state")
    .build()
    .await?;

On this definition-based path, only definition(...) or definition_path(...) is required. The other common fields have defaults:

Builder field	Default when omitted
`module_config`	Empty module and discovery configuration
`timeout`	30 seconds
`persistent_state`	Ephemeral session service with an auto-created temporary directory

The older mob_spec(...) path accepts a pre-built MobBootstrapSpec. It still requires explicit module configuration and timeout because the spec already owns its session service and construction policy:

use meerkat_mobkit::UnifiedRuntime;
use std::time::Duration;

let runtime = UnifiedRuntime::builder()
    .mob_spec(mob_bootstrap_spec)
    .module_config(mobkit_config)
    .timeout(Duration::from_secs(30))
    .build()
    .await?;

If neither definition source nor mob_spec is present, the builder returns MissingRequiredField(MobSpec). On the legacy path, omitting module_config or timeout returns the corresponding missing-field error.

Bootstrap sequence

Mob bootstrap

The Meerkat mob runtime starts first. Members are provisioned, wiring is established, and the mob reaches an operational state. If this fails, the builder returns UnifiedRuntimeBootstrapError::Mob.

Module startup

The supervisor attempts each discovered module and records its startup transitions. A module that exhausts its restart policy remains unloaded and emits a supervisor warning. A module-runtime initialization error or startup-thread panic rolls back the already-started mob before the builder returns an error.

Event merging

Agent and module events feed the unified event transport. Hosts read the public snapshot and subscription APIs rather than accessing the internal event vector.

Rollback on failure

If module-runtime initialization fails or its startup thread panics after the mob has already started, the runtime performs a coordinated rollback:

Attempt to shut down the mob runtime
If rollback succeeds, return the original module startup error
If rollback also fails, return ModuleStartupRollbackFailed containing both errors

This prevents orphaned mob processes when module-runtime bootstrap fails.

Runtime handle

After successful bootstrap, the builder returns a UnifiedRuntime handle with access to:

Method	Returns	Description
`mob_handle()`	`MobHandle`	Public Meerkat mob control handle
`mob_runtime()`	`&MobRuntime`	Underlying mob runtime owned by the unified runtime
`loaded_modules().await`	`Vec<String>`	Loaded module IDs
`module_events().await`	`Vec<EventEnvelope<UnifiedEvent>>`	Current module-runtime event snapshot
`subscribe_events(request).await`	`SubscribeResponse`	Bounded unified-event replay for the requested scope
`module_health_transitions().await`	`Vec<ModuleHealthTransition>`	Recorded module startup transitions
`set_error_hook(hook)`	`()`	Install the operational error hook after construction; see Error hook and logging

The raw MobkitRuntimeHandle is intentionally not exposed. Module operations such as routing, delivery, memory, and gating are methods on UnifiedRuntime.

Reconciliation

The reconciliation method depends on which authority model built the runtime. For a classic runtime without identity-first authority, pass the complete desired member specs:

let report: UnifiedRuntimeReconcileReport = runtime
    .reconcile(desired_specs)
    .await?;

desired_specs is a Vec<SpawnMemberSpec>. This path reconciles members, managed peer edges, and delivery routing. When the router module is loaded, it maintains route keys named mob.member.{member_id} with the notification channel. For an identity-first runtime, roster and topology providers own desired state. Refresh those providers through the identity authority instead:

let result = runtime.refresh_desired_topology().await?;

refresh_desired_topology returns Some(RestoreFlowResult) when identity-first authority is attached and None otherwise. Calling whole-mob reconcile on an identity-first runtime fails rather than bypassing that authority.

Shutdown

In the mob/module teardown sequence, graceful shutdown quiesces and stops the mob while its router and module dependencies remain alive. It then closes the event router and shuts down the modules:

let report: UnifiedRuntimeShutdownReport = runtime.shutdown().await;

The shutdown report includes status from both the mob and module runtimes.

HTTP composition and binding

build_reference_app_router is the public composition root for the health, console, Flow Editor, JSON-RPC, blob, and SSE surfaces. The router does not bind a socket by itself:

use meerkat_mobkit::gateway_composition::GatewayHttpBinding;
use std::sync::Arc;

let runtime = Arc::new(runtime);
let event_drain_task = Arc::clone(&runtime).spawn_event_drain_task();
let app = runtime.build_reference_app_router(decisions);
let binding = GatewayHttpBinding::bind_loopback().await?;
let http_base_url = binding.http_base_url();
let http_server = binding.serve(app);

println!("reference app: {http_base_url}/console");

GatewayHttpBinding::bind_loopback selects an available port on 127.0.0.1. The shipping gateways return that concrete http_base_url during initialization. Embedders may instead serve the returned Axum router on their own listener. A host-owned Axum server must keep event_drain_task running so mob events reach console SSE and event-log consumers. During ordered shutdown, drain http_server, abort the event-drain task, and then shut down the runtime. Key routes in the reference app include:

Route	Method	Description
`/healthz`	GET	Runtime health summary
`/console`	GET	Admin console HTML
`/console/assets/console-app.js`	GET	React bundle
`/console/assets/console-app.css`	GET	Stylesheet
`/console/experience`	GET	Experience metadata JSON
`/console/modules`	GET	Loaded modules JSON
`/console/identities`	GET	Identity records
`/console/timeline`	GET	Replayable timeline page
`/console/timeline/stream`	GET	Timeline replay/live SSE
`/console/identity/{identity}/stream`	GET	Identity-filtered timeline SSE
`/console/rpc`	POST	Console JSON-RPC
`/console/rpc/multipart`	POST	Multipart console JSON-RPC
`/blobs/{blob_id}`	GET	Blob/image retrieval
`/agents/{agent_id}/events`	GET	Per-agent event SSE
`/mob/events`	GET	Mob-merged event SSE
`/mobkit/mob_events/stream`	GET	Structural mob-events SSE
`/flow-editor`	GET	Visual mobpack editor
`/flow-editor/rpc`	POST	Mobpack-authoring JSON-RPC

Error hook and logging

Failures the runtime detects after bootstrap (a member spawn that failed, a checkpoint that did not persist, a stalled actor loop) have no caller to return to. They are emitted as typed ErrorEvents through the error hook, registered with UnifiedRuntimeBuilder::on_error or, for hosts that construct the runtime through UnifiedRuntime::bootstrap, with UnifiedRuntime::set_error_hook. The hook is fire-and-forget: it runs on a detached task, and a slow or failing hook cannot break the runtime. An attached identity-first runtime receives the same hook.

use meerkat_mobkit::{ErrorEvent, ErrorHook, UnifiedRuntime};
use std::sync::Arc;

let hook: ErrorHook = Arc::new(|event: ErrorEvent| {
    Box::pin(async move {
        match event {
            ErrorEvent::SpawnFailure { member_id, error, .. } => page(&member_id, &error).await,
            ErrorEvent::ActorLoopRecovered { .. } => {}
            other => alert(&other).await,
        }
    })
});

let runtime = UnifiedRuntime::builder()
    .definition_path("config/mob.toml")
    .on_error(hook)
    .build()
    .await?;

ErrorEvent is #[non_exhaustive] and serializes with an internal category tag in snake_case, so match with a wildcard arm and branch on category on the wire. The variants and where the runtime fires them:

`category`	Fields	Fired when
`spawn_failure`	`member_id`, `profile`, `error`	a member spawn fails
`reconcile_incomplete`	`failures`, `skipped`	`reconcile_edges` finished with failed or skipped edges
`checkpoint_failure`	`session_id`, `error`	the session store’s periodic checkpoint/GC failed
`compaction_persistence_rejected`	`identity`, `session_id`, `error`, `preserved_history?`, `attempted_entries?`	a member’s compaction handoff was refused durably; `preserved_history` says whether the member can still send
`actor_loop_stalled`	`probe_waited_secs`, `detail`, `stall_id?`, `prior_resolved_stalls?`	the actor-loop probe’s round trip went unanswered past its budget
`actor_loop_recovered`	`stall_id`, `stalled_for_secs`	that stall resolved; the one non-failure variant, correlated by `stall_id`
`host_loop_crash`	`member_id`, `error`	a member emitted `run_failed` during event drain
`rediscover_failure`	`error`	`rediscover` failed
`event_log_flush_failure`	`error`	the operational event log could not flush
`identity_materialization_failure`	`identity`, `initiator?`, `operation`, `error`	identity-first hydration skipped a member
`mob_stop_proceeded_without_interrupt`	`waited_ms`, `member?`, `error`	teardown gave up waiting for runtime-attach readiness and continued without interrupting the member

Default sink (since 0.8.18). Every ErrorEvent is also logged through tracing whether or not a hook is registered: at ERROR, or at INFO for actor_loop_recovered, with a hook_registered field. A host that never registers a hook additionally gets one WARN,

no error hook is registered, so runtime error events reach logs only; register one with UnifiedRuntimeBuilder::on_error (or UnifiedRuntime::set_error_hook) to route them to paging

: build() emits it at startup when on_error was never called, and the fire path emits it once per process for hosts that bootstrap directly and install no hook. Gateways. rpc_gateway installs a hook that forwards every event to the SDK as the mobkit/on_error JSON-RPC notification (params are the serialized event); register a receiver with Python .on_error(callback) or TypeScript .onError(callback). Because that forwarding hook is always installed, the absent-hook WARN never fires for SDK hosts; an SDK host that registers no callback sees only the gateway’s stderr ERROR line, which the SDKs inherit by default (see Gateway logging). mobkit_gateway installs no hook and relies on the log sink alone. Both binaries install a stderr tracing subscriber whose default filter is warn,meerkat_mobkit=info,<binary>=info; RUST_LOG overrides it. Rust library hosts must install a tracing subscriber before build(). meerkat-mobkit emits tracing events and installs no subscriber of its own. The default sink and the absent-hook notice are both tracing lines, so a host without a subscriber sees neither the failures nor the hint that nobody is listening. The minimum is a stderr subscriber honouring RUST_LOG:

tracing_subscriber::fmt()
    .with_env_filter(
        tracing_subscriber::EnvFilter::try_from_default_env()
            .unwrap_or_else(|_| tracing_subscriber::EnvFilter::new("warn,meerkat_mobkit=info")),
    )
    .with_writer(std::io::stderr)
    .init();

Error hierarchy

Error	Cause
`UnifiedRuntimeBuilderError::MissingRequiredField`	No definition/spec, or a required legacy-path field is missing
`UnifiedRuntimeBootstrapError::Mob`	Meerkat mob failed to start
`UnifiedRuntimeBootstrapError::Module`	MobKit module-runtime initialization failed
`UnifiedRuntimeBootstrapError::ModuleStartupThreadPanicked`	Module startup thread panicked
`UnifiedRuntimeBootstrapError::ModuleStartupRollbackFailed`	Both startup and rollback failed

Getting started

Core concepts

Guides

Unified Runtime

Builder paths

Bootstrap sequence

Rollback on failure

Runtime handle

Reconciliation

Shutdown

HTTP composition and binding

Error hook and logging

Error hierarchy

See also

​Builder paths

​Bootstrap sequence

​Rollback on failure

​Runtime handle

​Reconciliation

​Shutdown

​HTTP composition and binding

​Error hook and logging

​Error hierarchy

​See also

Builder paths

Bootstrap sequence

Rollback on failure

Runtime handle

Reconciliation

Shutdown

HTTP composition and binding

Error hook and logging

Error hierarchy

See also