> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rkat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Service setup

> Configure named spaces, credentials, processing approvals, source lifecycles and recovery.

Recall is an optional, independently operated knowledge service. Each request
selects one named space. Embedded storage, the compatible SurrealDB backend,
standalone deployment, graph records, wiki workflows and guarded source lifecycle
operations remain available. Installing an archive does not provision credentials,
approve model use, initialize a database or start a service.

## Install and initialize

Use the [installation guide](/recall/installation), retain the whole verified version
directory and configure its absolute `skills` directory as `PIPELINE_SKILLS_DIR`.
Service templates are supplied under `service/` in native archives. Create private
state, data, recovery, secrets, model asset and log directories for the service
account. Keep the recovery journal outside data and backup directories that can
be replaced during restore. Configure `migrate` and `serve` with the same backend,
namespace and database. Run the explicit `meerkat-recall migrate` command before
starting a new persistent service; `serve` verifies schema rather than silently
initializing it. See each command's `--help` for the chosen backend's settings.

Remote mode needs the corresponding patched SurrealDB server and capability
check. A matching upstream version number alone does not establish compatibility.
Its server binary is a separate artifact. The embedded managed service persists
acknowledged mutations using its configured durable storage destination.

## Select the runtime configuration

Recall 0.9.1 adds explicit TOML selection for managed services and migrations:

```sh theme={null}
meerkat-recall migrate --config /etc/meerkat-recall/runtime.toml
meerkat-recall serve --config /etc/meerkat-recall/runtime.toml
```

The command option takes precedence over `ELEPHANT_CONFIG`, which takes
precedence over the existing nearest-Git `.elephant/elephant_config.toml`
discovery. An explicitly selected missing or malformed file is an error; it
does not fall back to environment-only settings. These are `serve` and `migrate`
options, not global options. The legacy standalone worker remains environment
configured.

Relative config paths and relative data paths are resolved from the process
working directory. Use absolute paths for deployment data, snapshots and model
assets. Loading TOML selects its database, endpoint and model settings; ordinary
environment equivalents do not replace those fields. Snapshot-load and typed
SurrealDB timeout overrides retain their existing behavior.
`PIPELINE_AGENTIC_MCP_SPACE_ID` can override the configured discovery space.
API bind settings, signing/authentication settings, provider credentials and
`MCP_BEARER_TOKEN` remain separate environment settings.

## Credentials and space selection

The local CLI creates signing material and tokens without a running API:

```sh theme={null}
meerkat-recall service-token bootstrap --secret-file /run/recall/signing.key
meerkat-recall service-token issue --secret-file /run/recall/signing.key \
  --subject recall-operator --global-admin --expires-in 900 \
  --scope manage:grant --scope manage:provider \
  --scope read:space_config --scope write:space_config --scope read:status \
  --output /run/recall/operator.token
meerkat-recall service-token issue --secret-file /run/recall/signing.key \
  --subject recall-client --space research --space planning --expires-in 86400 \
  --scope read:source --scope write:source --scope read:doc:raw \
  --scope read:doc:metadata --scope write:doc --scope read:status \
  --output /run/recall/client.token
```

These are templates, not issued credentials. New files have private permissions;
the commands refuse existing outputs. Set `JWT_SECRET_FILE` to the signing file
and do not also set `JWT_SECRET`. Match configured issuer/audience requirements.
Normal tokens have a 31-day maximum lifetime; global management tokens have a
one-hour maximum. Global management credentials cannot perform ordinary data
access by selecting a space.

HTTP and MCP use `Authorization: Bearer <token>`. Path requests select their
named space. MCP, readiness and operation polling use `X-Space-Id: research`.
If both path and header select a space, they must agree exactly. A multi-space
token does not select the first listed space or create a default. The signed
space ceiling is intersected with the current local grant on each dispatch.

Create each space using `POST /v1/spaces`, with the matching `X-Space-Id`, before
granting the service subject access. Use `PUT /v1/spaces/research/grants/recall-client`
with an explicitly authorized management token and a JSON body such as:

```json theme={null}
{"active":true,"scopes":["read:source","write:source","read:doc:raw","read:doc:metadata","write:doc","read:status"],"expected_generation":null}
```

Record the returned generation. Later updates and `active:false` revocation use
that exact generation as `expected_generation`. Token scopes and grant scopes
both constrain access. Graph and wiki operations need their own explicit scopes.
Wildcard scopes do not manufacture `manage:grant` or `manage:provider` authority.
Revocation denies subsequent admissions; source-backed standing work also checks
its original persisted authority at its guarded storage boundaries. This does
not promise cancellation of every in-flight network request. There is no
individual JWT revocation list. Use distinct subjects for independently revocable
credentials. Signing-key replacement invalidates tokens under the old key.

Classification and subjects are separate from operation scopes. Explicit
`clearances` bind exact policy IDs, levels, labels and handling permissions.
Omitted or empty clearances retain the compatibility clearance fallback; they
are not a deny-all representation. An omitted subject allowlist is unrestricted
by subject, while an explicitly empty allowlist remains restrictive. Restricted
callers need complete authorized subject lineage. Never mark subjects complete
or drop handling caveats to bypass authorization. Metadata labels confer no
security authority. The issuer defines policy and audience conventions.

## Create a classified first space

Global administration tokens carry no custom policy clearance. To bootstrap a
classified named space, create an explicit clearance file and use the separate
short-lived named-space bootstrap profile:

```json theme={null}
[{"policy_id":"research-policy","max_level":3,"labels":[],"handling_allow":[]}]
```

```sh theme={null}
meerkat-recall service-token issue --secret-file /run/recall/signing.key \
  --subject research-bootstrap --bootstrap-space research --expires-in 900 \
  --scope read:space_config --scope write:space_config \
  --scope manage:grant --scope manage:provider \
  --clearances-file /private/recall/research-clearances.json \
  --output /private/recall/research-bootstrap.token
```

Choose policy values authorized for your deployment and match the server's
issuer/audience. With this bearer and `X-Space-Id: research`, create the space
using `POST /v1/spaces` with its exact name and `security` envelope, then create
its service grants. A bootstrap token is restricted to the named space's HTTP
space/grant administration; it cannot access data, MCP or job-producing routes.
Issue ordinary `--space research` tokens with matching explicit clearances for
the service subjects afterwards. Retire the bootstrap token after setup; its
maximum lifetime is one hour. Neither token issuance nor grant creation approves
provider work by itself.

## Configure extractor MCP access

The managed extractor calls Recall through `/mcp/agent`. Give it a separate
read-only identity for exactly the spaces it serves. This example uses `research`
and `planning`; repeat `--space` only for authorized spaces and provide explicit
clearances for each space's policy in the private clearance file:

```sh theme={null}
meerkat-recall service-token issue --secret-file /run/recall/signing.key \
  --subject recall-extractor --space research --space planning \
  --read-only --expires-in 86400 \
  --scope read:entity --scope read:predicate --scope read:doc:metadata \
  --scope read:doc:raw --scope read:source --scope read:assertion \
  --clearances-file /private/recall/extractor-clearances.json \
  --output /private/recall/extractor.token
```

Using the authorized management credential for each space, create an active
`recall-extractor` grant at `PUT /v1/spaces/{space}/grants/recall-extractor`:

```json theme={null}
{"active":true,"scopes":["read:entity","read:predicate","read:doc:metadata","read:doc:raw","read:source","read:assertion"],"expected_generation":null}
```

Read each grant back and retain its generation. Both token and grant must include
any additional read scopes required by your selected tools. `read:predicate` is
required for predicate discovery. Keep ordinary client credentials separate;
global management and named-space bootstrap tokens cannot serve as extractor
data credentials.

Supply the token value privately to the service or worker as `MCP_BEARER_TOKEN`
using its protected environment configuration. The variable takes the bearer,
not a token-file path. Arrange rotation before expiry. The extractor supplies the
current `X-Space-Id` on its MCP requests, so the same worker subject needs a
current grant in every selected space.

Set the extractor URL to the reachable listener plus `/mcp/agent`. For example,
a listener on port `3100` needs `http://127.0.0.1:3100/mcp/agent`; changing
`ELEPHANT_PORT` does not change the default extractor URL. In config-backed
bootstrap mode, pass `extractor_mcp_url` to `initialize_runtime` or
`configure_runtime` and verify it with `get_runtime_config`. Alternatively,
`PIPELINE_AGENTIC_MCP_URL` configures the environment-backed runtime; setting it
before launch selects configured mode instead of the unconfigured bootstrap
path. If maintenance is enabled, its separate `PIPELINE_MAINTENANCE_MCP_URL`
should reach `/mcp/maintenance` on the intended service.

Before admitting sources, connect an authenticated MCP client to `/mcp/agent`
with the worker bearer and each space's `X-Space-Id`. Complete the protocol
handshake and inspect `tools/list`: `get_doc`, `get_doc_chunk`, `get_entity`,
`search_entities`, `list_predicates`, `get_predicate` and `get_source` must be
available. Initialization or listener readiness alone does not establish this
worker authority. See [MCP transport and authentication](/recall/mcp/overview).

## Approve processing

A space configuration's `processing.connections` selects an account, provider,
endpoint and credential reference; `profiles` selects immutable model versions,
dimensions and preprocessing; `bindings` approves particular operations. A
remote `credential_ref` is an exact environment-variable name, without an `env:`
prefix. The reference is not the secret. Do not put secrets in URLs or metadata.

Source PUT uses Graph processing. Compatible approved `query_embedding` and
`document_embedding` bindings are required, together with `extraction` and
independent `support_verification` for semantic candidates. Summaries, media,
disambiguation, reindex, truth maintenance and wiki planner/writer/verifier
workflows require their respective approvals when selected. Optional operations
do not authorize themselves. Read current space configuration before replacing
it and preserve storage-owned index/reindex fields. Profiles are immutable;
changing model semantics means choosing a new profile. Populated-space profile
activation uses the staged reindex and explicit activation endpoints.

Governed FastEmbed takes a `file:///absolute/assets-root` endpoint and a
`sha256:<manifest-byte-digest>` model version. Use the bundled
`provision-fastembed` helper with independently selected ONNX and tokenizer files.
It needs Python 3.12+, copies selected files and never downloads model weights.
Retain model origin and license evidence. The runtime verifies the pinned files;
missing or changed bytes prevent use. Model names and installed caches do not
grant approval. Remote account health and successful inference are separate
from local configuration availability.

## Long documents and embedding profiles

For imports with documents above the selected embedding model's input limit,
choose the existing `text-full-v2` preprocessing profile explicitly. It segments
the complete input and pools segment vectors; it does not discard the suffix.
`text-v1` retains its original whole-input representation and refuses oversized
inputs with `embedding_input_too_large`. Never truncate documents or change the
preprocessing label on existing vectors to make an import pass.

Preprocessing belongs to each space's approved `processing.profiles` entry,
not `runtime.toml` or a global environment override. Read the current space with
`GET /v1/spaces/{space}`. Add a new immutable profile ID using the actual approved
connection, model, version and dimensions. For example, this profile shape uses
an operator-approved OpenAI connection and deployment identity:

```json theme={null}
{"id":"embedding-full-v2","connection_id":"approved-openai","model":"text-embedding-3-large","model_version":"operator-approved-version","dimensions":3072,"preprocessing":"text-full-v2"}
```

The example is a template; `model_version` records operator configuration and
does not independently pin a remote provider deployment. Preserve existing
profiles, bindings, generations and authority when submitting the complete
updated `processing` object through `PUT /v1/spaces/{space}`. This requires
`read:space_config`, `write:space_config`, explicit `manage:provider` and the
space's record/current-grant authority.

For a fresh empty space, bind both `query_embedding` and `document_embedding`
to the new profile and set `active_embedding_profile_id` to that profile. For a
populated space, leave its active profile and those bindings unchanged, add an
approved `reindex_embedding` binding to the new profile, then:

1. Read `GET /v1/spaces/{space}/processing/reindex` and retain its current
   `processing_generation`.
2. Call `POST /v1/spaces/{space}/processing/reindex/start` with
   `target_profile_id` and that `expected_processing_generation`.
3. Poll the reindex status until `pending_reindex.complete` is true. Retain the
   actual `reindex_id` and latest `processing_generation`. Concurrent content
   changes may require another pass; incomplete coverage cannot be activated.
4. Call `POST /v1/spaces/{space}/processing/reindex/activate` with the actual
   `reindex_id` and latest `expected_processing_generation`.

Activation updates the active profile and existing approved query/document
binding profile IDs together, while retaining revocations. Confirm the profile
and `embedding_execution_recipes` preprocessing in named-space status. Reindexing
uses the configured provider and may incur charges. After correcting the profile,
resubmit terminal failed sources using the [source retry contract](#operator-retries).

## Inspect budget before an import

From Recall 0.9.1, `GET /api/v1/budget` returns effective limits, current usage
and remaining headroom for the space selected by `X-Space-Id`. Use a named-space
operator token with `read:space_config`, its current same-space grant and the
SpaceConfig record's clearance. A status-only or bootstrap token cannot inspect
this endpoint.

```sh theme={null}
curl --fail --silent --show-error "$RECALL_URL/api/v1/budget" \
  -H "Authorization: Bearer $RECALL_OPERATOR_TOKEN" \
  -H 'X-Space-Id: research'
```

Inspect `runtime.effective`, `runtime.managed_workers_running` and `today.graph`
before bulk admission. `today.graph` and `today.retrieval` include `limit_units`,
`used_units`, `remaining_units`, `limit_items`, `used_items` and
`remaining_items`; `today.resets_at` gives the next UTC midnight. Usage includes
outstanding reservations plus reconciled work, including failed provider work.
The current guard wins over ledger aggregation, so reservations are not double
counted. Headroom is an advisory read, not a reservation or an atomic promise
across concurrent imports. Exhausted headroom is reported as zero, while
`used_units` and `used_items` retain any overrun.

The legacy names `daily_euros_*` and `spent_graph` are **abstract accounting
units, not provider invoices**: measured provider tokens reconcile at one unit
per million tokens. Reservations are estimates, and missing usage is not
fabricated. Do not multiply a total that includes reservations into a claim of
measured tokens or use it as a currency bill. Items count admitted work items,
not documents; one source can consume several graph items and attempts.
The older `GET /api/v1/budget/today` keeps its ledger-only totals and cannot
establish effective limits or reservation-aware headroom.

Budget policy is configured for the deployment but enforced independently for
each named space and UTC day. It is not a shared global account or a SpaceConfig
budget override. `runtime.configured` shows TOML values and `runtime.effective`
shows their managed-worker interpretation. Omitted optional item/token/time
caps fall back to worker defaults; they do not disable the cap. The managed
worker does not load `budget.concurrent_workers`; use the documented work-queue
concurrency settings. The endpoint cannot attest settings of separately launched
workers, which must be configured consistently.

To change limits for a TOML-managed deployment, edit the selected file's
`[budget]` section and restart gracefully, or use the authorized global MCP
`configure_runtime` tool with fields such as `budget_daily_euros_graph` and
`budget_daily_items_graph`. Global runtime configuration requires `write:system`
and an unrestricted global principal; ordinary space administration does not
change the deployment's budget. Use `get_runtime_config` and the selected-space
budget endpoint to verify the result. Environment-only workers use
`PIPELINE_BUDGET_*` at startup; those variables do not override a loaded TOML
budget. See [all budget settings](/recall/operations/deployment#pipeline-budget-control).

## Admit and reconcile sources

Use `PUT /v1/spaces/{space}/sources/{external_id}` with a new opaque `revision`,
nonempty UTF-8 `text`, `media_type` (`text/plain` or `text/markdown`), required
`observed_at` timestamp, and the
source's authorized metadata, labels, classification and subject lineage.
Replacement uses the exact current `expected_generation`. Percent-encode path
components once. Source text is limited to 2,000,000 decoded bytes. Revisions
bind immutable text, labels and semantic/security metadata; choose a new revision
when those values change. Metadata keys prefixed by `_` are reserved.

An optional `Idempotency-Key` identifies the immutable request. Identical replay
returns its original operation. A reused revision with different immutable
payload is a conflict. On `conflict_generation`, fetch the current source head
and reconcile before submitting a replacement. Ordinary retry must not request
reactivation or recreation after erasure.

Acceptance is durable intent. Public operation states are `pending`, `running`,
`complete`, `failed_retryable` and `failed_final`. Poll the returned operation,
then read the current source head. Earlier completed operations retain their
historical outcome after replacement or withdrawal; they do not promise current
visibility. Unfinished superseded work reports `superseded`.

Operation polling exposes a fixed `error_code` when work is waiting:
`awaits_budget` records an actual typed budget refusal; `awaits_source_bundle`
and `awaits_source_index` identify the corresponding missing publication
prerequisite. A finalizer poll does not clear a child's recorded budget wait.
Inspect effective headroom and current work before changing limits or retrying;
these wait codes do not claim that the source has completed or that every
missing bundle has the same cause.

Search using `POST /v1/spaces/{space}/search` or MCP `search_scoped`. Bind
`source_revision_constraints` to the desired external IDs and revisions when
checking publication. Responses include `results`, `coverage` and `partial`.
Each result carries evidence with source/revision, half-open UTF-8 byte spans
and exact quotes. Rendered evidence separately binds the immutable rendered
revision. `complete` coverage and zero results can mean no match; pending,
stale, missing or withdrawn coverage does not establish that conclusion.

Explicit ranking modes are `lexical`, `semantic` and `hybrid`. Lexical needs no
model. Semantic requires actual indexed evidence and does not silently fall back
to lexical results. Filters cannot widen the caller's policy or subject ceiling.
Promoted assertions, entities, relationships and events require valid immutable
evidence and independently verified support. Interpretation without support
cannot become approved knowledge. Grounding is rechecked at publication.

`GET /v1/spaces/{space}/sources` provides authorized heads with an opaque cursor.
It is a bounded reconciliation inventory, not a snapshot or lossless change feed.
Store external-to-internal source identity mappings at admission. After physical
erasure, inventory does not reconstruct erased external identifiers.

## Operator retries

A failed work item can be retried through `POST /v1/work-items/{id}/retry` or the
MCP `retry_work_item` tool. Issue a separate operator token with both
`read:work_item` and `write:work_item`, plus a current same-space grant and the
record's required ABAC authority. Do not add write scopes to the read-only
extraction worker token. The item must be Failed and its producer/dependency
state must permit `retry_authorized`; operator authority does not bypass those
guards. Poll the original item and operation after an uncertain response before
requesting another retry.

A source operation already marked `failed_final` retains its terminal receipt;
retrying its old work item or replaying its old idempotency key does not create a
new source attempt. Correct the provider/profile/budget issue first and read the
current source head for its exact generation. For a document originally submitted
to `POST /v1/sources/ingest`, resubmit the **original importer payload** with a
new `intake.revision_id`, a new `intake.idempotency_key` and
`intake.expected_generation` equal to that current generation. Keep the original
normalized `name`/document fields, or the original legacy `source` and `doc`
objects, plus the requested ingest mode. Preserve complete text, media, metadata,
classification, subjects and extraction guidance unless an authorized change is
intended. The intake object is separate from document metadata:

```json theme={null}
{"intake":{"revision_id":"fresh-recovery-revision","idempotency_key":"fresh-recovery-request","expected_generation":"actual-current-source-generation"}}
```

Add that object to the complete original request; it is not a standalone ingest
body. Poll the newly returned operation to completion and retain the old failed
operation for audit. A changed generation is a conflict to reconcile, not
permission to overwrite another writer. This flow uses current source and
document scopes, record authority and grants; it does not require giving write
access to the extractor worker.

Source GET responses strip the private `_manual_doc` template even when raw-source
read permission is present. Do not round-trip a GET response into PUT and expect
document reprocessing. The retained `PUT /v1/sources/{id}` API is usable when the
caller already holds the complete intended original Source payload/template.
Its Source fields belong at the top level beside the fresh `intake` object, not
inside a `source` property. The versioned source PUT similarly requires its
complete original public input. Reprocess jobs wait for unpublished desired
heads, so they are not a substitute for fresh intake of failed, unpublished
documents.

Work-queue transient retries default to 10 attempts in Recall 0.9.1; the existing
`PIPELINE_WORK_TRANSIENT_RETRY_MAX_ATTEMPTS` override remains authoritative.
Exponential backoff retains its 10-second base and 120-second cap. This does
not raise a model/provider repair budget, change storage's generic retry policy
or make terminal failures retryable.

## Recover an existing vector index

The Recall 0.9.1 engine correction persists identical-vector membership changes
and prevents filtered-out records from consuming result slots. It does not
reconstruct membership already missing from an existing index. Restarting or
writing the same vector again is not proof of recovery: unchanged indexed values
can skip index maintenance. Remote storage requires the corrected paired server
or the corresponding external engine repair; a Recall client upgrade alone is
insufficient.

For a source stuck before publication, use the original document intake and fresh
revision/key/current-generation contract under [operator retries](#operator-retries).
This retains the source identity and creates new physical document, node and
vector identities under the new publication fence. A previous accepted,
processing or retryable operation becomes superseded; an old `failed_final`
receipt remains unchanged. Observe the **new** operation until it completes and
verify the published source head. HTTP 202 records acceptance, not recovery.

Governed embedding reindex skips unpublished source nodes and is not a repair
for their missing index memberships. Its [staged profile migration](#long-documents-and-embedding-profiles)
requires a distinct approved target and complete published-data coverage before
activation. Do not change profiles merely to try to unstick an unpublished
source. Recall does not expose a general embedded-index rebuild command; do not
edit live database files or bypass publication guards.

Fresh processing may call configured providers and consume budget. Check
[headroom](#inspect-budget-before-an-import) before resubmission. Existing calls
and receipts are neither replayed nor refunded by a new intake, and upgrading
does not by itself establish that affected sources have recovered.

## Readiness, shutdown and recovery

`GET /v1/health` is process liveness. Authenticated `GET /v1/readiness` with
`read:status` and an explicit selected space reports storage, worker, recovery
and semantic ingestion readiness. `GET /v1/spaces/{space}/status` adds local
provider/configuration availability. Use a separate scoped status token with
`read:status` in both its signed ceiling and its current selected-space grant.
Normal ABAC and expiry checks still apply.

From Recall 0.9.1, extractor discovery requires the seven canonical read tools;
a successful HTTP response or status-only `tools/list` is insufficient. For a
multi-space worker token, set `extractor.readiness_space_id` in TOML or
`PIPELINE_AGENTIC_MCP_SPACE_ID` in the environment to select one discovery space.
Actual extraction always selects its work item's space and rechecks authority;
one successful discovery does not grant access to other spaces. Legacy
single-space tokens retain their existing selection behavior.

For the exact agent route on this listener's literal loopback address,
named-space status also verifies the worker token using this service's signing
configuration and current grants. Remote or reverse-proxied MCP endpoints have
their own authority: local diagnostics explicitly leave those keys and grants
unchecked and direct the operator to the remote endpoint. Global discovery
still establishes only the selected discovery space's tools. Readers with
`read:space_config` receive fixed diagnostic actions; status-only callers do not
receive token, subject, grant contents or private endpoint details.
These checks do not call paid providers or prove that a particular revision is
searchable. An API-only role can be ready while workers are unavailable.
Pending operations remain pending truthfully.

Use the graceful shutdown path. Managed workers and requests drain before the
storage barrier; a failed drain or flush is not a successful persistence receipt.
Retain the store ownership file so overlapping writers cannot bypass exclusion.

Retract and erase use the current generation and explicit lifecycle routes:
`POST /v1/spaces/{space}/sources/{external_id}:retract` or `:erase`. Reactivation
requires `reactivate:true`; recreation after erasure requires
`recreate_after_erasure:true`. Both use the current tombstone generation.
Erasure completion requires physical source-derived deletion. Apply a bounded
retention policy to backups; hiding records from query results is insufficient.
Keep the independent durable erasure journal and restore barrier outside replaced
backup/data trees. Restore validates and replays that authority before reopening
admission. See the [snapshot guide](https://github.com/lukacf/meerkat-recall/blob/d71722482183a802fcaeaaf41cb04ef7d16f4b85/docs/snapshot-durability.md) and
[embedded commit guide](https://github.com/lukacf/meerkat-recall/blob/d71722482183a802fcaeaaf41cb04ef7d16f4b85/docs/embedded-durable-commits.md).

MCP retains the protocol handshake and `tools/call` envelope. Inspect `tools/list`
for the selected profile (`/mcp`, `/mcp/agent`, `/mcp/maintenance`, `/mcp/full`).
Retained ingest, graph, events and wiki tools keep their own argument and result
contracts. Use typed `{error_code,message,retryable}` errors rather than parsing
message text; a bare routing 404 is not a typed unknown-source response.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.