Skip to main content
Recall is an optional, independently operated knowledge service. Each request selects one named space. Embedded storage, the compatible SurrealDB backend, standalone deployment, graph records, wiki workflows and guarded source lifecycle operations remain available. Installing an archive does not provision credentials, approve model use, initialize a database or start a service.

​
Install and initialize

Use the installation guide, retain the whole verified version directory and configure its absolute skills directory as PIPELINE_SKILLS_DIR. Service templates are supplied under service/ in native archives. Create private state, data, recovery, secrets, model asset and log directories for the service account. Keep the recovery journal outside data and backup directories that can be replaced during restore. Configure migrate and serve with the same backend, namespace and database. Run the explicit meerkat-recall migrate command before starting a new persistent service; serve verifies schema rather than silently initializing it. See each command’s --help for the chosen backend’s settings. Remote mode needs the corresponding patched SurrealDB server and capability check. A matching upstream version number alone does not establish compatibility. Its server binary is a separate artifact. The embedded managed service persists acknowledged mutations using its configured durable storage destination.

​
Select the runtime configuration

Recall 0.9.1 adds explicit TOML selection for managed services and migrations:
The command option takes precedence over ELEPHANT_CONFIG, which takes precedence over the existing nearest-Git .elephant/elephant_config.toml discovery. An explicitly selected missing or malformed file is an error; it does not fall back to environment-only settings. These are serve and migrate options, not global options. The legacy standalone worker remains environment configured. Relative config paths and relative data paths are resolved from the process working directory. Use absolute paths for deployment data, snapshots and model assets. Loading TOML selects its database, endpoint and model settings; ordinary environment equivalents do not replace those fields. Snapshot-load and typed SurrealDB timeout overrides retain their existing behavior. PIPELINE_AGENTIC_MCP_SPACE_ID can override the configured discovery space. API bind settings, signing/authentication settings, provider credentials and MCP_BEARER_TOKEN remain separate environment settings.

​
Credentials and space selection

The local CLI creates signing material and tokens without a running API:
These are templates, not issued credentials. New files have private permissions; the commands refuse existing outputs. Set JWT_SECRET_FILE to the signing file and do not also set JWT_SECRET. Match configured issuer/audience requirements. Normal tokens have a 31-day maximum lifetime; global management tokens have a one-hour maximum. Global management credentials cannot perform ordinary data access by selecting a space. HTTP and MCP use Authorization: Bearer <token>. Path requests select their named space. MCP, readiness and operation polling use X-Space-Id: research. If both path and header select a space, they must agree exactly. A multi-space token does not select the first listed space or create a default. The signed space ceiling is intersected with the current local grant on each dispatch. Create each space using POST /v1/spaces, with the matching X-Space-Id, before granting the service subject access. Use PUT /v1/spaces/research/grants/recall-client with an explicitly authorized management token and a JSON body such as:
Record the returned generation. Later updates and active:false revocation use that exact generation as expected_generation. Token scopes and grant scopes both constrain access. Graph and wiki operations need their own explicit scopes. Wildcard scopes do not manufacture manage:grant or manage:provider authority. Revocation denies subsequent admissions; source-backed standing work also checks its original persisted authority at its guarded storage boundaries. This does not promise cancellation of every in-flight network request. There is no individual JWT revocation list. Use distinct subjects for independently revocable credentials. Signing-key replacement invalidates tokens under the old key. Classification and subjects are separate from operation scopes. Explicit clearances bind exact policy IDs, levels, labels and handling permissions. Omitted or empty clearances retain the compatibility clearance fallback; they are not a deny-all representation. An omitted subject allowlist is unrestricted by subject, while an explicitly empty allowlist remains restrictive. Restricted callers need complete authorized subject lineage. Never mark subjects complete or drop handling caveats to bypass authorization. Metadata labels confer no security authority. The issuer defines policy and audience conventions.

​
Create a classified first space

Global administration tokens carry no custom policy clearance. To bootstrap a classified named space, create an explicit clearance file and use the separate short-lived named-space bootstrap profile:
Choose policy values authorized for your deployment and match the server’s issuer/audience. With this bearer and X-Space-Id: research, create the space using POST /v1/spaces with its exact name and security envelope, then create its service grants. A bootstrap token is restricted to the named space’s HTTP space/grant administration; it cannot access data, MCP or job-producing routes. Issue ordinary --space research tokens with matching explicit clearances for the service subjects afterwards. Retire the bootstrap token after setup; its maximum lifetime is one hour. Neither token issuance nor grant creation approves provider work by itself.

​
Configure extractor MCP access

The managed extractor calls Recall through /mcp/agent. Give it a separate read-only identity for exactly the spaces it serves. This example uses research and planning; repeat --space only for authorized spaces and provide explicit clearances for each space’s policy in the private clearance file:
Using the authorized management credential for each space, create an active recall-extractor grant at PUT /v1/spaces/{space}/grants/recall-extractor:
Read each grant back and retain its generation. Both token and grant must include any additional read scopes required by your selected tools. read:predicate is required for predicate discovery. Keep ordinary client credentials separate; global management and named-space bootstrap tokens cannot serve as extractor data credentials. Supply the token value privately to the service or worker as MCP_BEARER_TOKEN using its protected environment configuration. The variable takes the bearer, not a token-file path. Arrange rotation before expiry. The extractor supplies the current X-Space-Id on its MCP requests, so the same worker subject needs a current grant in every selected space. Set the extractor URL to the reachable listener plus /mcp/agent. For example, a listener on port 3100 needs http://127.0.0.1:3100/mcp/agent; changing ELEPHANT_PORT does not change the default extractor URL. In config-backed bootstrap mode, pass extractor_mcp_url to initialize_runtime or configure_runtime and verify it with get_runtime_config. Alternatively, PIPELINE_AGENTIC_MCP_URL configures the environment-backed runtime; setting it before launch selects configured mode instead of the unconfigured bootstrap path. If maintenance is enabled, its separate PIPELINE_MAINTENANCE_MCP_URL should reach /mcp/maintenance on the intended service. Before admitting sources, connect an authenticated MCP client to /mcp/agent with the worker bearer and each space’s X-Space-Id. Complete the protocol handshake and inspect tools/list: get_doc, get_doc_chunk, get_entity, search_entities, list_predicates, get_predicate and get_source must be available. Initialization or listener readiness alone does not establish this worker authority. See MCP transport and authentication.

​
Approve processing

A space configuration’s processing.connections selects an account, provider, endpoint and credential reference; profiles selects immutable model versions, dimensions and preprocessing; bindings approves particular operations. A remote credential_ref is an exact environment-variable name, without an env: prefix. The reference is not the secret. Do not put secrets in URLs or metadata. Source PUT uses Graph processing. Compatible approved query_embedding and document_embedding bindings are required, together with extraction and independent support_verification for semantic candidates. Summaries, media, disambiguation, reindex, truth maintenance and wiki planner/writer/verifier workflows require their respective approvals when selected. Optional operations do not authorize themselves. Read current space configuration before replacing it and preserve storage-owned index/reindex fields. Profiles are immutable; changing model semantics means choosing a new profile. Populated-space profile activation uses the staged reindex and explicit activation endpoints. Governed FastEmbed takes a file:///absolute/assets-root endpoint and a sha256:<manifest-byte-digest> model version. Use the bundled provision-fastembed helper with independently selected ONNX and tokenizer files. It needs Python 3.12+, copies selected files and never downloads model weights. Retain model origin and license evidence. The runtime verifies the pinned files; missing or changed bytes prevent use. Model names and installed caches do not grant approval. Remote account health and successful inference are separate from local configuration availability.

​
Long documents and embedding profiles

For imports with documents above the selected embedding model’s input limit, choose the existing text-full-v2 preprocessing profile explicitly. It segments the complete input and pools segment vectors; it does not discard the suffix. text-v1 retains its original whole-input representation and refuses oversized inputs with embedding_input_too_large. Never truncate documents or change the preprocessing label on existing vectors to make an import pass. Preprocessing belongs to each space’s approved processing.profiles entry, not runtime.toml or a global environment override. Read the current space with GET /v1/spaces/{space}. Add a new immutable profile ID using the actual approved connection, model, version and dimensions. For example, this profile shape uses an operator-approved OpenAI connection and deployment identity:
The example is a template; model_version records operator configuration and does not independently pin a remote provider deployment. Preserve existing profiles, bindings, generations and authority when submitting the complete updated processing object through PUT /v1/spaces/{space}. This requires read:space_config, write:space_config, explicit manage:provider and the space’s record/current-grant authority. For a fresh empty space, bind both query_embedding and document_embedding to the new profile and set active_embedding_profile_id to that profile. For a populated space, leave its active profile and those bindings unchanged, add an approved reindex_embedding binding to the new profile, then:
  1. Read GET /v1/spaces/{space}/processing/reindex and retain its current processing_generation.
  2. Call POST /v1/spaces/{space}/processing/reindex/start with target_profile_id and that expected_processing_generation.
  3. Poll the reindex status until pending_reindex.complete is true. Retain the actual reindex_id and latest processing_generation. Concurrent content changes may require another pass; incomplete coverage cannot be activated.
  4. Call POST /v1/spaces/{space}/processing/reindex/activate with the actual reindex_id and latest expected_processing_generation.
Activation updates the active profile and existing approved query/document binding profile IDs together, while retaining revocations. Confirm the profile and embedding_execution_recipes preprocessing in named-space status. Reindexing uses the configured provider and may incur charges. After correcting the profile, resubmit terminal failed sources using the source retry contract.

​
Inspect budget before an import

From Recall 0.9.1, GET /api/v1/budget returns effective limits, current usage and remaining headroom for the space selected by X-Space-Id. Use a named-space operator token with read:space_config, its current same-space grant and the SpaceConfig record’s clearance. A status-only or bootstrap token cannot inspect this endpoint.
Inspect runtime.effective, runtime.managed_workers_running and today.graph before bulk admission. today.graph and today.retrieval include limit_units, used_units, remaining_units, limit_items, used_items and remaining_items; today.resets_at gives the next UTC midnight. Usage includes outstanding reservations plus reconciled work, including failed provider work. The current guard wins over ledger aggregation, so reservations are not double counted. Headroom is an advisory read, not a reservation or an atomic promise across concurrent imports. Exhausted headroom is reported as zero, while used_units and used_items retain any overrun. The legacy names daily_euros_* and spent_graph are abstract accounting units, not provider invoices: measured provider tokens reconcile at one unit per million tokens. Reservations are estimates, and missing usage is not fabricated. Do not multiply a total that includes reservations into a claim of measured tokens or use it as a currency bill. Items count admitted work items, not documents; one source can consume several graph items and attempts. The older GET /api/v1/budget/today keeps its ledger-only totals and cannot establish effective limits or reservation-aware headroom. Budget policy is configured for the deployment but enforced independently for each named space and UTC day. It is not a shared global account or a SpaceConfig budget override. runtime.configured shows TOML values and runtime.effective shows their managed-worker interpretation. Omitted optional item/token/time caps fall back to worker defaults; they do not disable the cap. The managed worker does not load budget.concurrent_workers; use the documented work-queue concurrency settings. The endpoint cannot attest settings of separately launched workers, which must be configured consistently. To change limits for a TOML-managed deployment, edit the selected file’s [budget] section and restart gracefully, or use the authorized global MCP configure_runtime tool with fields such as budget_daily_euros_graph and budget_daily_items_graph. Global runtime configuration requires write:system and an unrestricted global principal; ordinary space administration does not change the deployment’s budget. Use get_runtime_config and the selected-space budget endpoint to verify the result. Environment-only workers use PIPELINE_BUDGET_* at startup; those variables do not override a loaded TOML budget. See all budget settings.

​
Admit and reconcile sources

Use PUT /v1/spaces/{space}/sources/{external_id} with a new opaque revision, nonempty UTF-8 text, media_type (text/plain or text/markdown), required observed_at timestamp, and the source’s authorized metadata, labels, classification and subject lineage. Replacement uses the exact current expected_generation. Percent-encode path components once. Source text is limited to 2,000,000 decoded bytes. Revisions bind immutable text, labels and semantic/security metadata; choose a new revision when those values change. Metadata keys prefixed by _ are reserved. An optional Idempotency-Key identifies the immutable request. Identical replay returns its original operation. A reused revision with different immutable payload is a conflict. On conflict_generation, fetch the current source head and reconcile before submitting a replacement. Ordinary retry must not request reactivation or recreation after erasure. Acceptance is durable intent. Public operation states are pending, running, complete, failed_retryable and failed_final. Poll the returned operation, then read the current source head. Earlier completed operations retain their historical outcome after replacement or withdrawal; they do not promise current visibility. Unfinished superseded work reports superseded. Operation polling exposes a fixed error_code when work is waiting: awaits_budget records an actual typed budget refusal; awaits_source_bundle and awaits_source_index identify the corresponding missing publication prerequisite. A finalizer poll does not clear a child’s recorded budget wait. Inspect effective headroom and current work before changing limits or retrying; these wait codes do not claim that the source has completed or that every missing bundle has the same cause. Search using POST /v1/spaces/{space}/search or MCP search_scoped. Bind source_revision_constraints to the desired external IDs and revisions when checking publication. Responses include results, coverage and partial. Each result carries evidence with source/revision, half-open UTF-8 byte spans and exact quotes. Rendered evidence separately binds the immutable rendered revision. complete coverage and zero results can mean no match; pending, stale, missing or withdrawn coverage does not establish that conclusion. Explicit ranking modes are lexical, semantic and hybrid. Lexical needs no model. Semantic requires actual indexed evidence and does not silently fall back to lexical results. Filters cannot widen the caller’s policy or subject ceiling. Promoted assertions, entities, relationships and events require valid immutable evidence and independently verified support. Interpretation without support cannot become approved knowledge. Grounding is rechecked at publication. GET /v1/spaces/{space}/sources provides authorized heads with an opaque cursor. It is a bounded reconciliation inventory, not a snapshot or lossless change feed. Store external-to-internal source identity mappings at admission. After physical erasure, inventory does not reconstruct erased external identifiers.

​
Operator retries

A failed work item can be retried through POST /v1/work-items/{id}/retry or the MCP retry_work_item tool. Issue a separate operator token with both read:work_item and write:work_item, plus a current same-space grant and the record’s required ABAC authority. Do not add write scopes to the read-only extraction worker token. The item must be Failed and its producer/dependency state must permit retry_authorized; operator authority does not bypass those guards. Poll the original item and operation after an uncertain response before requesting another retry. A source operation already marked failed_final retains its terminal receipt; retrying its old work item or replaying its old idempotency key does not create a new source attempt. Correct the provider/profile/budget issue first and read the current source head for its exact generation. For a document originally submitted to POST /v1/sources/ingest, resubmit the original importer payload with a new intake.revision_id, a new intake.idempotency_key and intake.expected_generation equal to that current generation. Keep the original normalized name/document fields, or the original legacy source and doc objects, plus the requested ingest mode. Preserve complete text, media, metadata, classification, subjects and extraction guidance unless an authorized change is intended. The intake object is separate from document metadata:
Add that object to the complete original request; it is not a standalone ingest body. Poll the newly returned operation to completion and retain the old failed operation for audit. A changed generation is a conflict to reconcile, not permission to overwrite another writer. This flow uses current source and document scopes, record authority and grants; it does not require giving write access to the extractor worker. Source GET responses strip the private _manual_doc template even when raw-source read permission is present. Do not round-trip a GET response into PUT and expect document reprocessing. The retained PUT /v1/sources/{id} API is usable when the caller already holds the complete intended original Source payload/template. Its Source fields belong at the top level beside the fresh intake object, not inside a source property. The versioned source PUT similarly requires its complete original public input. Reprocess jobs wait for unpublished desired heads, so they are not a substitute for fresh intake of failed, unpublished documents. Work-queue transient retries default to 10 attempts in Recall 0.9.1; the existing PIPELINE_WORK_TRANSIENT_RETRY_MAX_ATTEMPTS override remains authoritative. Exponential backoff retains its 10-second base and 120-second cap. This does not raise a model/provider repair budget, change storage’s generic retry policy or make terminal failures retryable.

​
Recover an existing vector index

The Recall 0.9.1 engine correction persists identical-vector membership changes and prevents filtered-out records from consuming result slots. It does not reconstruct membership already missing from an existing index. Restarting or writing the same vector again is not proof of recovery: unchanged indexed values can skip index maintenance. Remote storage requires the corrected paired server or the corresponding external engine repair; a Recall client upgrade alone is insufficient. For a source stuck before publication, use the original document intake and fresh revision/key/current-generation contract under operator retries. This retains the source identity and creates new physical document, node and vector identities under the new publication fence. A previous accepted, processing or retryable operation becomes superseded; an old failed_final receipt remains unchanged. Observe the new operation until it completes and verify the published source head. HTTP 202 records acceptance, not recovery. Governed embedding reindex skips unpublished source nodes and is not a repair for their missing index memberships. Its staged profile migration requires a distinct approved target and complete published-data coverage before activation. Do not change profiles merely to try to unstick an unpublished source. Recall does not expose a general embedded-index rebuild command; do not edit live database files or bypass publication guards. Fresh processing may call configured providers and consume budget. Check headroom before resubmission. Existing calls and receipts are neither replayed nor refunded by a new intake, and upgrading does not by itself establish that affected sources have recovered.

​
Readiness, shutdown and recovery

GET /v1/health is process liveness. Authenticated GET /v1/readiness with read:status and an explicit selected space reports storage, worker, recovery and semantic ingestion readiness. GET /v1/spaces/{space}/status adds local provider/configuration availability. Use a separate scoped status token with read:status in both its signed ceiling and its current selected-space grant. Normal ABAC and expiry checks still apply. From Recall 0.9.1, extractor discovery requires the seven canonical read tools; a successful HTTP response or status-only tools/list is insufficient. For a multi-space worker token, set extractor.readiness_space_id in TOML or PIPELINE_AGENTIC_MCP_SPACE_ID in the environment to select one discovery space. Actual extraction always selects its work item’s space and rechecks authority; one successful discovery does not grant access to other spaces. Legacy single-space tokens retain their existing selection behavior. For the exact agent route on this listener’s literal loopback address, named-space status also verifies the worker token using this service’s signing configuration and current grants. Remote or reverse-proxied MCP endpoints have their own authority: local diagnostics explicitly leave those keys and grants unchecked and direct the operator to the remote endpoint. Global discovery still establishes only the selected discovery space’s tools. Readers with read:space_config receive fixed diagnostic actions; status-only callers do not receive token, subject, grant contents or private endpoint details. These checks do not call paid providers or prove that a particular revision is searchable. An API-only role can be ready while workers are unavailable. Pending operations remain pending truthfully. Use the graceful shutdown path. Managed workers and requests drain before the storage barrier; a failed drain or flush is not a successful persistence receipt. Retain the store ownership file so overlapping writers cannot bypass exclusion. Retract and erase use the current generation and explicit lifecycle routes: POST /v1/spaces/{space}/sources/{external_id}:retract or :erase. Reactivation requires reactivate:true; recreation after erasure requires recreate_after_erasure:true. Both use the current tombstone generation. Erasure completion requires physical source-derived deletion. Apply a bounded retention policy to backups; hiding records from query results is insufficient. Keep the independent durable erasure journal and restore barrier outside replaced backup/data trees. Restore validates and replays that authority before reopening admission. See the snapshot guide and embedded commit guide. MCP retains the protocol handshake and tools/call envelope. Inspect tools/list for the selected profile (/mcp, /mcp/agent, /mcp/maintenance, /mcp/full). Retained ingest, graph, events and wiki tools keep their own argument and result contracts. Use typed {error_code,message,retryable} errors rather than parsing message text; a bare routing 404 is not a typed unknown-source response.