> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rkat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Development

> Build commands, cargo aliases, pre-commit hooks, testing tiers, and workspace lints

## Key Dependencies

Before diving into the codebase, it helps to understand the two external systems Meerkat Recall depends on:

<Accordion title="Storage backends - SurrealDB and elephant-db">
  [SurrealDB](https://surrealdb.com) is a multi-model database written in Rust. It handles document storage, graph traversals, full-text search, and vector similarity search in a single engine.

  **Why Meerkat Recall uses it:** It is the default persistent backend. Instead of running Postgres + Neo4j + a separate vector store, Meerkat Recall can use one SurrealDB deployment for documents, graph, and vectors.

  **`elephant-db`:** Meerkat Recall also has an embedded in-memory backend for lightweight local/runtime use and the WASM-oriented kernel path. It supports graph traversal, lexical/vector search, and JSON snapshots, but it is not a general SurrealQL replacement.

  **Development setup:** `docker compose up` starts a SurrealDB instance on port 8000. Embedded mode is enabled with `ELEPHANT_DB=elephant-db`.

  **Gotchas:** SurrealDB's native Rust driver has specific serialization behaviors that cause subtle bugs if you don't follow the [four mandatory patterns](/recall/operations/surrealdb-patterns). Those rules apply to the Surreal-backed path; embedded mode bypasses the Surreal boundary entirely.
</Accordion>

<Accordion title="Meerkat - the LLM agent harness">
  [Meerkat](https://github.com/lukacf/meerkat) is a modular LLM agent harness built in Rust. It provides schema-constrained output, tool dispatch, budget enforcement, and streaming for LLM interactions.

  **Where Meerkat Recall uses it:**

  * **Extraction pipeline** (`crates/pipeline/src/extraction/agentic_extractor.rs`) - the agentic extractor uses Meerkat with an assertion output schema, bounded retries, graph query tools, and budget limits. Meerkat Recall parses and validates the result before canonical writes; a failed or malformed completion still fails extraction.
  * **Truth maintenance** (`crates/pipeline/src/truth_maintenance/adjudicator.rs`) - conflict adjudication calls the LLM through Meerkat structured output with the competing claim values in view, returning a strict-schema verdict (corroborate, single winner, coexist, supersede, needs review). Entity-reuse disambiguation at promotion uses the same pattern.
  * **Living Wiki** - Meerkat powers separate writer and verifier calls. Meerkat Recall
    serves its MCP endpoint using the Rust `rmcp` transport.

  **Feature flag:** The Meerkat-based extractor is behind the `meerkat-extraction` Cargo feature. When disabled, Meerkat Recall no longer performs degraded semantic extraction; the managed runtime reports `semantic_ingest_ready=false`, ingest tools are hidden, and extraction work fails loudly if forced to run without a real extractor.
</Accordion>

<Note>
  The primary runtime is now MCP-first and config-backed. Meerkat Recall can start in bootstrap mode with no storage configured yet, expose `get_runtime_status` and runtime setup tools, and only expose semantic ingest once storage, workers, and the extractor are all ready.
</Note>

Two practical nuances:

* if you set runtime env vars like `ELEPHANT_DB` or `SURREALDB_URL`, Meerkat Recall starts in configured mode rather than bootstrap mode
* self-referential extractor URLs such as `PIPELINE_AGENTIC_MCP_URL=http://127.0.0.1:3000/mcp/agent` are supported; Meerkat Recall retries extractor readiness after the listener binds, so clients can poll `get_runtime_status` instead of issuing a manual reconnect

The example assumes the listener is on port `3000`. Match the extractor URL to
the actual listener and configure `MCP_BEARER_TOKEN` with current named-space
authority, including predicate reads. Listener retries cannot repair an incorrect
endpoint or token. Follow [extractor MCP setup](/recall/operations/service-setup#configure-extractor-mcp-access).

## Repository Structure

```
bin/           # Binaries
  elephant/             # Unified CLI (`meerkat-recall serve`, `worker`, `migrate`, `snapshot`, `status`)
  elephant-api/         # Internal HTTP API library crate
  elephant-pipeline/    # Internal worker runtime crate

crates/        # Core libraries
  elephant-db/  # Embedded in-memory graph/vector storage kernel
  types/        # Domain types, DTOs, record IDs, validation
  storage/      # SurrealDB repositories and connection pooling
  policy/       # ABAC policy engine and redaction
  identity/     # Identity key management and merge strategy
  knowledge-views/ # Derived views, wiki editorial compilation, validation
  pipeline/     # Ingestion, extraction, TM, freshness, reprocess workers
  embeddings/   # Vector provider abstraction
  mcp/          # MCP protocol handlers and tool registry
  outbox/       # Event outbox for reliable messaging
  extensions/   # Extension manifest and reference handling
  migrations/   # SurrealQL migration runner

tests/         # Test suites
  scenarios/    # E2E scenario tests (S1.x-S4.x, M1-M10)
  e2e/          # E2E test implementations
  calibration/  # Prompt calibration harness (golden-case LLM scoring)
  dev_readiness/ # Development readiness verification
```

## Build Commands

Use the repo wrapper for Rust commands so each checkout/worktree gets isolated Cargo state:

```bash theme={null}
./scripts/repo-cargo check -p meerkat-recall-bin -p elephant
./scripts/repo-cargo test -p elephant-storage
```

Cargo aliases are still available through the wrapper and are defined in `.cargo/config.toml`:

| Alias | Command | Description |
| - | - | - |
| `./scripts/repo-cargo b` | `./scripts/repo-cargo build --workspace` | Build all workspace crates |
| `./scripts/repo-cargo t` | `./scripts/repo-cargo test --workspace` | Run non-ignored workspace tests; broader than the pure unit lane |
| `./scripts/repo-cargo c` | `./scripts/repo-cargo clippy --workspace --all-targets --all-features -- -D warnings` | Lint with warnings as errors |
| `./scripts/repo-cargo f` | `./scripts/repo-cargo fmt --all` | Format all code |
| `./scripts/repo-cargo doc` | `./scripts/repo-cargo doc --workspace --no-deps` | Generate documentation |
| `./scripts/repo-cargo integration` | `./scripts/repo-cargo test --workspace -- --ignored` | Run all ignored tests; may invoke live paid providers |
| `./scripts/repo-cargo test-e2e` | See `.cargo/config.toml` for provider-test exclusions | Deterministic ignored E2E suite |
| `./scripts/repo-cargo test-e2e-model` | Exact local-model constructor regression | Requires explicitly provisioned model endpoint and manifest digest |
| `./scripts/repo-cargo test-e2e-live` | `./scripts/repo-cargo test -p elephant-e2e live_llm -- --ignored --nocapture --test-threads=1` | Opt-in Alice and Wizard live extraction scenarios |
| `./scripts/repo-cargo test-api` | `./scripts/repo-cargo test --package elephant-api` | API tests only |
| `./scripts/repo-cargo test-pipeline` | `./scripts/repo-cargo test --package elephant-pipeline` | Pipeline tests only |
| `./scripts/repo-cargo test-storage` | `./scripts/repo-cargo test --package elephant-storage` | Storage tests only |

### CI scope and resource tests

CI classifies exact Git changes before selecting Rust work. Only modifications
of reviewed reader-prose files qualify for the short path. Code examples, inline
code, MDX expressions, frontmatter, new or renamed files, scripts, workflows,
manifests, configuration and runtime sources use the full path. The living-wiki
page is embedded in an executable and always uses the full path. An unavailable
or invalid comparison fails classification; required gates cannot pass on that
failure. Manual runs always execute the full gates.

Required check names remain stable. A prose-only check explicitly reports that
Rust execution is not applicable; secret scanning, dependency auditing,
container configuration and reader-documentation validation still run. Rust
caches use the paths reported by `scripts/repo-cargo --print-env`, separated by
job and compiler/manifest/environment identity.

`scripts/ci/ignored_test_lanes.json` records every ignored Rust test. The
ordinary deterministic E2E command retains its four large storage scale tests.
Paid provider, calibration and real-media tests remain explicit opt-in work.
The local-model constructor regression is a separate fixture-backed command:

```bash theme={null}
export RECALL_TEST_FASTEMBED_ENDPOINT=file:///absolute/path/to/model-assets
export RECALL_TEST_FASTEMBED_MANIFEST_SHA256=YOUR_VERIFIED_MANIFEST_SHA256
make test-e2e-model
```

The endpoint is the root containing the immutable manifest-digest directory
created by `scripts/release/provision-fastembed`. The test refuses absent inputs;
it does not download a model or return a false success. In CI, a dedicated setup
step downloads the exact MIT-licensed BAAI model revision and file hashes in
`scripts/ci/model_fixture.json`, verifies every file, then invokes that same
provisioner. The original test checks actual initialization reuse, independent
request receipts and revocation; deterministic tests run separately in the same
E2E job. Updating a model pin or test inventory requires the full CI path.

### Running Specific Tests

<CodeGroup>
  ```bash Single test theme={null}
  ./scripts/repo-cargo test test_name -- --exact
  ```

  ```bash Package tests theme={null}
  ./scripts/repo-cargo test -p elephant-api
  ./scripts/repo-cargo test -p elephant-pipeline
  ./scripts/repo-cargo test -p elephant-storage
  ```

  ```bash Scenario tests theme={null}
  ./scripts/repo-cargo test -p elephant-scenarios -- --nocapture
  ```
</CodeGroup>

### Makefile Targets

The `Makefile` is the single source of truth for local hooks, CI, and release commands:

| Target | Description |
| - | - |
| `make test` | Fast workspace test lane |
| `make test-unit` | Workspace unit/binary lane |
| `make test-all` | Full all-features workspace test lane |
| `make hook-test` | Pre-push unit lane |
| `make hook-test-all` | Pre-push full lane |
| `make test-scenarios` | Scenario tests only |
| `make hook-lint` | Run clippy |
| `make fmt` | Fix formatting |
| `make verify-surface` | Verify active surface and legacy drift |
| `make verify-version-parity` | Verify crate/workspace version alignment |
| `make ci` | Full CI pipeline |
| `make ci-smoke` | Faster smoke CI lane |
| `make release-preflight` | Release validation lane |
| `make publish-dry-run` | Cargo package dry-run for publishable crates |
| `make install-hooks` | Install pre-commit hooks |

## Pre-Commit Hooks

Install hooks with:

```bash theme={null}
pip install pre-commit
pre-commit install
pre-commit install --hook-type pre-push
```

Or use the Makefile shortcut: `make install-hooks`.

### On Commit

The commit-stage hook is intentionally small:

| Hook | Description |
| - | - |
| `cargo-fmt-fix` | Auto-format Rust files via `./scripts/repo-cargo fmt --all` |

### On Push

| Hook | Description |
| - | - |
| `gitleaks` | Secret detection |
| file hygiene hooks | whitespace, EOF, YAML/TOML, merge conflict, key detection |
| `cargo-fmt` | `make fmt-check` |
| `cargo-clippy` | `make hook-lint` |
| `verify-surface` | `make verify-surface` |
| `cargo-test` | workspace unit gate via `scripts/pre-push-unit.sh` |

Heavy all-features tests, docs, E2E, WASM, and audit run in CI/release lanes rather than every local pre-push.

## Testing Tiers

Meerkat Recall uses four testing tiers with increasing scope:

<Steps>
  <Step title="Unit tests">
    Domain and repository behavior tested using `#[cfg(test)]` modules within each crate. Run the unit/binary lane with `make test-unit`.

    No external dependencies required. Tests run entirely in-process.
  </Step>

  <Step title="Integration tests">
    Storage tests use the in-process `kv-mem` backend. Some expensive and end-to-end
    tests are marked `#[ignore]`; `make test-e2e` selects deterministic ignored
    checks. `./scripts/repo-cargo integration` runs all ignored tests and may
    include provider-backed scenarios.

    Each test creates a unique namespace/database pair for isolation:

    ```rust theme={null}
    let config = StorageConfig {
        url: "mem://".to_string(),
        namespace: format!("ns_{}", uuid),
        database: format!("db_{}", uuid),
        ..StorageConfig::default()
    };
    ```
  </Step>

  <Step title="E2E Scenarios">
    System-level tests verifying cross-cutting invariants. See the [Scenarios](/recall/operations/scenarios) page for the full test map.

    Run with:

    ```bash theme={null}
    ./scripts/repo-cargo test -p elephant-scenarios -- --nocapture
    ```
  </Step>

  <Step title="Prompt calibration">
    Golden-case suites in `tests/calibration` (package `elephant-calibration`) score the LLM prompt entrypoints -- truth-maintenance adjudication and entity disambiguation -- per field against labeled expected verdicts, writing `results.json` for prompt-version diffing. The real-LLM runners are `#[ignore]`d and skip cleanly when `ANTHROPIC_API_KEY` is unset. See `tests/calibration/README.md`.

    ```bash theme={null}
    ANTHROPIC_API_KEY=... ./scripts/repo-cargo test -p elephant-calibration -- --ignored --nocapture
    ```
  </Step>
</Steps>

## Workspace Lints

The workspace enforces strict linting rules:

| Lint | Level | Description |
| - | - | - |
| `unsafe_code` | `forbid` | Unsafe code is completely prohibited |
| `unwrap_used` | `warn` | Prefer error handling over `.unwrap()` |
| `expect_used` | `warn` | Prefer error handling over `.expect()` |
| `panic` | `warn` | Avoid explicit panics |
| `todo` | `warn` | Avoid the `todo!()` macro in production code |
| Clippy `pedantic` | enabled | Pedantic lints for code quality |
| Clippy `nursery` | enabled | Experimental lints for early issue detection |

All clippy warnings are treated as errors (`-D warnings`) in CI and pre-push hooks.

## Local Development

```bash theme={null}
# Start the development database and apply the loopback-only dev environment
docker compose -f docker-compose.dev.yml up -d
source .envrc
./scripts/repo-cargo run -p meerkat-recall-bin -- migrate

# Start Meerkat Recall API / MCP runtime
./scripts/repo-cargo run -p meerkat-recall-bin -- serve
```

This example sets `SKIP_AUTH=true` and binds to loopback for local development.
Use MCP `get_runtime_status` (or `GET /v1/runtime/status`) to inspect readiness.
Without storage environment variables or persisted configuration, Meerkat Recall starts
in bootstrap mode. Authenticated startup still requires a valid `JWT_SECRET`.
The preferred local project config lives at `.elephant/elephant_config.toml`.

For an isolated embedded demo:

```bash theme={null}
SKIP_AUTH=true ELEPHANT_HOST=127.0.0.1 ELEPHANT_DB=elephant-db \
  ./scripts/repo-cargo run -p meerkat-recall-bin -- serve
```

An optional bootstrap snapshot can be selected with
`ELEPHANT_DB_SNAPSHOT_LOAD=/tmp/elephant.json`.

The `.cargo/config.toml` sets default environment variables for development:

* `RUST_BACKTRACE=1` for full backtraces
* `RUST_LOG=info,elephant=debug` for useful log output

## Diagnose bulk ingestion

Before a bulk run, inspect the selected space's effective daily budget and
reservation-aware headroom through `GET /api/v1/budget`; see
[operator budget setup](/recall/operations/service-setup#inspect-budget-before-an-import).
For oversized inputs, preserve source bytes and choose a new approved
`text-full-v2` profile with the
[guarded reindex sequence](/recall/operations/service-setup#long-documents-and-embedding-profiles).
Do not relabel existing `text-v1` vectors or silently truncate input.

Authorized work-run metrics retain `custom.failure_diagnostic` with fixed
`kind`, `stage`, `code` and `action` fields. Worker warnings expose the matching
`failure_kind`, `failure_stage`, `failure_code` and `failure_action`. Use these
typed causes to distinguish model input/output limits, provider quota, MCP
transport, extraction budget and schema failures. A new diagnostic cannot
recover a cause already absent from a historical receipt. Keep raw source text,
provider error bodies and credentials out of shared reports.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.