Skip to main content

​
Storage Operations

Meerkat resolves durable realm state through one StorageLayout and one realm-level storage provider. The rkat storage commands operate below normal runtime scope so they can inspect or repair states that ordinary realm resolution correctly refuses. The three commands have deliberately different safety contracts: These commands never read, move, or report credential stores. A persistent rkat-rpc host also keeps approval records in approvals.json under its resolved persistence store path. That RPC-owned sidecar is not an eighth StorageProvider domain, and the current storage doctor, migration, and prune commands do not inventory, migrate, or delete it.

​
Select Roots And Realms

Without explicit roots, storage commands inspect two sweep candidates:
  • invocation-context-local <context-root>/.rkat/realms
  • the user-global Meerkat data root
Ordinary runtime resolution first walks up to the nearest ancestor containing .rkat, so run storage commands from that project root (or pass --root) when you want their local sweep to match the runtime’s project-local candidate. Use global --state-root or repeatable command --root to provide an explicit set. Once either is present, only those roots are read; ambient candidates are not added. Global --realm restricts work to one validated realm ID while still detecting twins across every swept root.
Storage commands dispatch before a model or session runtime is built, so runtime-only flags such as --default-model are rejected.

​
Read-Only Diagnosis

rkat storage doctor is safe against a live realm. It takes no realm lease or maintenance fence, never changes logical database content, and never creates a database or directory. A read-only SQLite connection may create or update the -wal/-shm sidecars required for a coherent read of a live WAL database. On truly read-only media, a WAL database without the required sidecars fails typed instead of falling back to an unsafe immutable read.
The report covers:
  • realm inventory by root and manifest/backend state
  • per-database meerkat_schema ledger domains and versions
  • the same realm ID materialized under more than one swept root
  • persisted-session structural decode failures and transcript footprint
  • dangling session-to-blob references
  • orphaned lock, lease, index, and registered backup artifacts
Exit status is 0 when there are no error-severity findings and 1 when errors are present. Warnings and informational findings alone do not make the command fail.
The CLI reaches the built-in disk migrator because it intentionally runs before an external-provider realm can be composed. External providers expose the same StorageMigrator::diagnose contract through their provider-native operator surface; an unknown external backend is report-only to this CLI.

​
Dry-Run And Apply Migrations

storage migrate is a dry run by default:
The dry run reports ledger baselines, path adoption, split-brain state, and deprecated leftovers without changing bytes. Per-mob databases under mobs/ remain report-only in migration v1. Apply only with the realm offline or after verifying all writers honor the shared maintenance fence:
Apply acquires the exclusive per-realm maintenance fence, waits for shared per-operation guards to drain, and then opens each store through its normal constructor so exact released predecessors migrate under their owning ledger domain. The default fence wait is 10 seconds; change it with --fence-wait-secs. Migration refuses when the fence cannot be acquired, a schema is from the future, the manifest is unreadable, durable identity is ambiguous, or input does not match a recognized predecessor. It never guesses a merge. Exit status is 0 for a clean report and 1 for an error or fail-closed refusal.

​
Resolve A Split-Brain Realm

If one realm ID exists under multiple swept roots, ordinary runtime resolution and migration both fail closed. First inspect the divergence report:
Choose the root whose copy should remain authoritative, then apply explicitly:
All copies are fenced and divergence is recomputed while those fences are held. The selected copy remains where it lies. Other copies are archived read-only with the registered *.pre-<version>-<timestamp> naming scheme. Meerkat does not synthesize or merge divergent WorkGraph event streams, schedule aggregates, or session histories. --adopt-root requires --apply and must name one of the swept roots that actually contains the realm.

​
Recover Storage From Before 0.8.10

Ordinary opens enforce the v0.8.10 compatibility floor. Current releases also provide an explicit, fenced bridge for authenticated historical SQLite schemas:
The bridge is apply-only and supported only when the realm manifest selects the sqlite backend. It verifies an exact known physical schema and durable evidence before importing, then normal strict migration continues under the same fence. Refusal has three distinct scopes:
  • Whole-bridge preflight: unsupported backends (JSONL, memory, or external providers), invalid manifests, malformed ledgers, and future-version evidence refuse before bridge imports or ledger stamping begin.
  • Per-domain transaction: an unauthenticated source schema or failed schema migration refuses that domain without committing its migration. Domains already committed remain committed; a later refusal does not roll them back.
  • Per-record admission: within an authenticated domain, healthy records can migrate and the domain ledger can be stamped even when other records cannot be carried forward. Those refused records remain byte-identical on disk and are reported individually.
Inspect both domain errors and the notes naming records left behind. An openable realm or completed domain does not prove that every historical session is resumable. Take an external backup or immutable generation snapshot before this recovery path. The bridge is not an authorization to repair by hand or invent missing authority.

​
Prune Registered Maintenance Artifacts

storage prune is unrelated to live realm deletion. Its dry run lists only recognized migration backups and index quarantines:
Apply deletion to artifacts at least 30 days old (the default):
Choose another minimum age explicitly:
Only *.pre-<version>-<timestamp> backups and *.corrupt-<timestamp> quarantines under the swept roots are candidates. The command never deletes a live realm or an unregistered filename. Use rkat realm delete or rkat realm prune for realm lifecycle. Exit status is 1 only when deletion/reporting errors occur.

​
Recover From A Durable Projection Halt

If appending a canonical session event fails, the persistent session path records a durable projection-halt marker and refuses replay or resume past the hole. There is currently no self-service command to clear or repair that marker, and storage doctor does not report it. Do not delete or edit it by hand. Stop writers, preserve the complete store for diagnostics, and restore an independently verified complete generation or escalate the store for provider-specific recovery.

​
Repair A WholeBlob Session Wedged On Its Audited Endpoint

Symptom: after a turn completes, completed_boundary_commit fails with Store read failed: WholeBlob body is not a valid current Session: live transcript does not preserve the graph-proved audited endpoint, every later send answers reload_required, and a reload fails on the same error at durable-tail recovery. The committed document’s live rows no longer match the final materialization of its audited transcript graph, so the current-envelope decoder refuses it on every read. Current writers refuse to mint such a document (the checkpoint fails instead, naming the first divergent row), and readers surface the typed SESSION_WHOLEBLOB_AUDITED_ENDPOINT_DIVERGENCE session error (resume hold audited_endpoint_divergence) so hosts can tell “needs repair” from I/O failure. For a session that is already wedged, repair it once:
  1. Stop or evict the member first. The service refuses while any live actor owns the session (a live checkpointer or a live registry entry), because that actor’s in-memory state, not the committed document, is then the authority. Under Meta, stop the member (or let the identity retire) and confirm no reload_member retry loop is running against it.
  2. Point rkat at the runtime database that holds the session. A Meta gateway on the default disk layout opens one runtime database directly, <state_dir>/runtime.sqlite (flat under the state directory; the <state_dir>/meta/ realm directory next to it holds only the meerkat-level jobs store and manifest pin), so name that file with --runtime-store and pass the full session UUID. A store that lives in a realm layout (<state_root>/<realm>/runtime.sqlite3, the shape a composite storage provider’s meerkat-level realm mobkit uses) is addressed with --state-root and --realm instead; the realm argument is the realm’s directory name under the realms root (ASCII letters, digits, - or _). The mob.<name> id recorded in a session’s metadata is a mob scope, not a store realm; passing it is refused with a message that says so. A Meta member parked with a session_repair_required hold carries the exact form for its own store in diagnose_command and apply_command. The command opens only that runtime database: it never creates a realm manifest or any other store, a missing database is a typed refusal naming the path it looked for, and a database pinned to the head-canonical profile is refused because this repair only applies to WholeBlob documents. A database whose runtime-store schema is older than the rkat binary is refused through a read-only preflight rather than migrated in place; open it once with the gateway that owns it (or a matching rkat) first. The only writes an open can make are SQLite’s own: the runtime.sqlite3.mfence lock sibling, the WAL sidecars, and the journal-mode conversion to WAL for a database not already in WAL mode. No --realm-backend flag is needed.
  3. Diagnose, then apply:
A diagnose report for the shape of a 2026-09-22 production incident reads, with counts only: "action": "would_reanchor", "divergence": {"kind": "live_prefix_diverges", "endpoint_row_count": 118, "live_row_count": 125, "first_divergent_row": 115}, "graph_edges_dropped": 8, one dropped compaction intent, and the base store revision and blob digest it read (base_blob_sha256, the digest of the committed WholeBlob document body; it is not an image blob id, see Blob digests in meerkat and Meta). The apply report repeats those facts with "action": "reanchored" and the new committed_store_revision; a second diagnose then reports "action": "no_repair_needed" with the decoded live_row_count. The expected divergence for that report is LivePrefixDiverges (live rows at least as many as the endpoint, a prefix row differing). A LiveShorterThanEndpoint divergence is the one shape where re-anchoring on the live rows drops content the audited endpoint proves existed; --apply refuses it unless you pass --accept-shorter, and the report then records both row counts as the audit trail of that decision. The repair keeps every message byte for byte. It removes the audited transcript graph, its rewrite prefix authority and the pending compaction projection intents from the document metadata, commits the result through the store’s compare-and-swap against the observed authority (a concurrent writer makes it report a conflict rather than overwrite), and leaves the store-owned compaction outbox row so the runtime’s ordinary finalization still indexes the discarded content. Audit history restarts at the next rewrite. The repair refuses when the live rows themselves do not decode, when the re-anchored document would change any row, or when a live actor still owns the session. Hosts embedding PersistentSessionService call repair_whole_blob_audited_endpoint(session_id, apply, accept_shorter) for the same procedure.

​
Blob Digests In Meerkat And Meta

Meerkat and Meta both render image blob ids as sha256:<64 lowercase hex digits>, and both feed a media type, a 0x00 separator and the image into SHA-256. They hash different representations of the image, so the same PNG carries a different id under each recipe. Never compare a meerkat content blob id with a Meta blob id: the ids are indistinguishable by shape. Which recipe a Meta-backed store applies depends on the entry point, not on the store. Every image that reaches a Meta store through meerkat’s BlobStore face (Base64BlobStoreAdapter::put_image, used by the gateway, the unified runtime and the console voice host) and every image uploaded through the console (meta/blob/upload, multipart image_upload placeholders) is addressed with meerkat’s content_blob_id and stored under the canonical media type, so meerkat’s read-back verification accepts it. Only blobs written through Meta’s native BinaryBlobStore::put_bytes (non-image meta/blob/* objects) carry the raw-bytes form, as do image blobs written by Meta builds that predate this rule. A Meta store resolves both forms through both faces, so references written before the rule keep hydrating. Why the same image differs: base64 text of a byte string is not that byte string, so the two SHA-256 inputs differ for every non-empty image. Meerkat chose the encoded form because an inline image in a transcript hydrates from its blob’s own base64 and the two representations must share one identity. Meta chose the decoded form because its byte-serving store never holds the base64 text. How to recognize which form you hold: the string does not tell you, the provenance does. An id that appears in a meerkat session document (an externalized image row, an audit edge, a hydration error, a BlobIdentityMismatch) is a meerkat content blob id, including when the store behind it is Meta’s. An id minted for a non-image object by put_bytes, or for an image by a Meta build that predates meerkat addressing, is a Meta blob id; both appear in blob object paths, /blobs/{id} URLs and meta/blob/* metadata. Meta’s local object store also still reads a legacy layout whose ids were minted over the base64 text without media-type canonicalization; those legacy ids equal meerkat’s only when the media type was already canonical. What is safe: compare meerkat ids with meerkat ids (an externalized row against its blob store entry, a hydration payload against its row) and Meta ids with Meta ids. Re-deriving one from the other requires the payload: decode the meerkat base64 and hash it under Meta’s recipe, or encode the Meta bytes and hash them under meerkat’s recipe. Do not hand-roll either derivation in new code; call the owning function. Unification of the two schemes onto one recipe is planned as its own release, with a digest migration for existing manifests and repair tooling that re-anchors stored ids. Until then the two schemes coexist as documented here.

​
Store-attested legacy references

A transcript can still hold an image reference in the Meta raw-bytes form, written by a Meta build that predates meerkat addressing. Meerkat’s recomputation cannot match such a reference, so a store may vouch for it: BlobStore::attest_address(blob_id, payload) returns BlobAddressAttestation::StoreAddress when the store recomputes its own address recipe over exactly the returned payload and gets blob_id. The default implementation returns Unattested, which keeps every gate fail-closed. An implementation must recompute from the payload; answering from a lookup, or attesting unconditionally, defeats the check. Two paths consult the attestation:
  • The durable fork preflight (which fork_off and council participant forks go through) verifies through verify_stored_image_blob_accepting_store_address and re-homes an attested reference: it stores the payload under meerkat’s content_blob_id and rewrites the reference in the child’s copy of the transcript only. The source transcript and its legacy object are left as they are.
  • Realtime image hydration accepts an attested reference.
An unattested mismatch still fails with BlobIdentityMismatch. A Meta store attests its legacy references once its blob adapter implements attest_address.

​
Operational Runbook

For an unreadable or refused realm:
  1. Stop writers if any mutation may be needed.
  2. Run storage doctor --json against an explicit root set and save the report with the deployment evidence.
  3. If the issue is split-brain, identify the authoritative generation from external provenance, then use --apply --adopt-root. Do not choose by newest timestamp alone.
  4. If the report identifies pre-v0.8.10 SQLite state, take a backup and use the explicit compatibility bridge.
  5. Run storage migrate as a dry run, then --apply only after the report is clean and the maintenance fence can be acquired.
  6. Run storage doctor again and boot the candidate normally.
  7. Prune registered backups only after the rollback window expires.

​
See Also