Development

Key Dependencies

Before diving into the codebase, it helps to understand the two external systems Meerkat Recall depends on:

Storage backends - SurrealDB and elephant-db

SurrealDB is a multi-model database written in Rust. It handles document storage, graph traversals, full-text search, and vector similarity search in a single engine.Why Meerkat Recall uses it: It is the default persistent backend. Instead of running Postgres + Neo4j + a separate vector store, Meerkat Recall can use one SurrealDB deployment for documents, graph, and vectors.elephant-db: Meerkat Recall also has an embedded in-memory backend for lightweight local/runtime use and the WASM-oriented kernel path. It supports graph traversal, lexical/vector search, and JSON snapshots, but it is not a general SurrealQL replacement.Development setup: docker compose up starts a SurrealDB instance on port 8000. Embedded mode is enabled with ELEPHANT_DB=elephant-db.Gotchas: SurrealDB’s native Rust driver has specific serialization behaviors that cause subtle bugs if you don’t follow the four mandatory patterns. Those rules apply to the Surreal-backed path; embedded mode bypasses the Surreal boundary entirely.

Meerkat - the LLM agent harness

Meerkat is a modular LLM agent harness built in Rust. It provides schema-constrained output, tool dispatch, budget enforcement, and streaming for LLM interactions.Where Meerkat Recall uses it:

Extraction pipeline (crates/pipeline/src/extraction/agentic_extractor.rs) - the agentic extractor uses Meerkat with an assertion output schema, bounded retries, graph query tools, and budget limits. Meerkat Recall parses and validates the result before canonical writes; a failed or malformed completion still fails extraction.
Truth maintenance (crates/pipeline/src/truth_maintenance/adjudicator.rs) - conflict adjudication calls the LLM through Meerkat structured output with the competing claim values in view, returning a strict-schema verdict (corroborate, single winner, coexist, supersede, needs review). Entity-reuse disambiguation at promotion uses the same pattern.
Living Wiki - Meerkat powers separate writer and verifier calls. Meerkat Recall serves its MCP endpoint using the Rust rmcp transport.

Feature flag: The Meerkat-based extractor is behind the meerkat-extraction Cargo feature. When disabled, Meerkat Recall no longer performs degraded semantic extraction; the managed runtime reports semantic_ingest_ready=false, ingest tools are hidden, and extraction work fails loudly if forced to run without a real extractor.

The primary runtime is now MCP-first and config-backed. Meerkat Recall can start in bootstrap mode with no storage configured yet, expose get_runtime_status and runtime setup tools, and only expose semantic ingest once storage, workers, and the extractor are all ready.

Two practical nuances:

if you set runtime env vars like ELEPHANT_DB or SURREALDB_URL, Meerkat Recall starts in configured mode rather than bootstrap mode
self-referential extractor URLs such as PIPELINE_AGENTIC_MCP_URL=http://127.0.0.1:3000/mcp/agent are supported; Meerkat Recall retries extractor readiness after the listener binds, so clients can poll get_runtime_status instead of issuing a manual reconnect

The example assumes the listener is on port 3000. Match the extractor URL to the actual listener and configure MCP_BEARER_TOKEN with current named-space authority, including predicate reads. Listener retries cannot repair an incorrect endpoint or token. Follow extractor MCP setup.

Repository Structure

bin/           # Binaries
  elephant/             # Unified CLI (`meerkat-recall serve`, `worker`, `migrate`, `snapshot`, `status`)
  elephant-api/         # Internal HTTP API library crate
  elephant-pipeline/    # Internal worker runtime crate

crates/        # Core libraries
  elephant-db/  # Embedded in-memory graph/vector storage kernel
  types/        # Domain types, DTOs, record IDs, validation
  storage/      # SurrealDB repositories and connection pooling
  policy/       # ABAC policy engine and redaction
  identity/     # Identity key management and merge strategy
  knowledge-views/ # Derived views, wiki editorial compilation, validation
  pipeline/     # Ingestion, extraction, TM, freshness, reprocess workers
  embeddings/   # Vector provider abstraction
  mcp/          # MCP protocol handlers and tool registry
  outbox/       # Event outbox for reliable messaging
  extensions/   # Extension manifest and reference handling
  migrations/   # SurrealQL migration runner

tests/         # Test suites
  scenarios/    # E2E scenario tests (S1.x-S4.x, M1-M10)
  e2e/          # E2E test implementations
  calibration/  # Prompt calibration harness (golden-case LLM scoring)
  dev_readiness/ # Development readiness verification

Build Commands

Use the repo wrapper for Rust commands so each checkout/worktree gets isolated Cargo state:

./scripts/repo-cargo check -p meerkat-recall-bin -p elephant
./scripts/repo-cargo test -p elephant-storage

Cargo aliases are still available through the wrapper and are defined in .cargo/config.toml:

Alias	Command	Description
`./scripts/repo-cargo b`	`./scripts/repo-cargo build --workspace`	Build all workspace crates
`./scripts/repo-cargo t`	`./scripts/repo-cargo test --workspace`	Run non-ignored workspace tests; broader than the pure unit lane
`./scripts/repo-cargo c`	`./scripts/repo-cargo clippy --workspace --all-targets --all-features -- -D warnings`	Lint with warnings as errors
`./scripts/repo-cargo f`	`./scripts/repo-cargo fmt --all`	Format all code
`./scripts/repo-cargo doc`	`./scripts/repo-cargo doc --workspace --no-deps`	Generate documentation
`./scripts/repo-cargo integration`	`./scripts/repo-cargo test --workspace -- --ignored`	Run all ignored tests; may invoke live paid providers
`./scripts/repo-cargo test-e2e`	See `.cargo/config.toml` for provider-test exclusions	Deterministic ignored E2E suite
`./scripts/repo-cargo test-e2e-model`	Exact local-model constructor regression	Requires explicitly provisioned model endpoint and manifest digest
`./scripts/repo-cargo test-e2e-live`	`./scripts/repo-cargo test -p elephant-e2e live_llm -- --ignored --nocapture --test-threads=1`	Opt-in Alice and Wizard live extraction scenarios
`./scripts/repo-cargo test-api`	`./scripts/repo-cargo test --package elephant-api`	API tests only
`./scripts/repo-cargo test-pipeline`	`./scripts/repo-cargo test --package elephant-pipeline`	Pipeline tests only
`./scripts/repo-cargo test-storage`	`./scripts/repo-cargo test --package elephant-storage`	Storage tests only

CI scope and resource tests

CI classifies exact Git changes before selecting Rust work. Only modifications of reviewed reader-prose files qualify for the short path. Code examples, inline code, MDX expressions, frontmatter, new or renamed files, scripts, workflows, manifests, configuration and runtime sources use the full path. The living-wiki page is embedded in an executable and always uses the full path. An unavailable or invalid comparison fails classification; required gates cannot pass on that failure. Manual runs always execute the full gates. Required check names remain stable. A prose-only check explicitly reports that Rust execution is not applicable; secret scanning, dependency auditing, container configuration and reader-documentation validation still run. Rust caches use the paths reported by scripts/repo-cargo --print-env, separated by job and compiler/manifest/environment identity. scripts/ci/ignored_test_lanes.json records every ignored Rust test. The ordinary deterministic E2E command retains its four large storage scale tests. Paid provider, calibration and real-media tests remain explicit opt-in work. The local-model constructor regression is a separate fixture-backed command:

export RECALL_TEST_FASTEMBED_ENDPOINT=file:///absolute/path/to/model-assets
export RECALL_TEST_FASTEMBED_MANIFEST_SHA256=YOUR_VERIFIED_MANIFEST_SHA256
make test-e2e-model

The endpoint is the root containing the immutable manifest-digest directory created by scripts/release/provision-fastembed. The test refuses absent inputs; it does not download a model or return a false success. In CI, a dedicated setup step downloads the exact MIT-licensed BAAI model revision and file hashes in scripts/ci/model_fixture.json, verifies every file, then invokes that same provisioner. The original test checks actual initialization reuse, independent request receipts and revocation; deterministic tests run separately in the same E2E job. Updating a model pin or test inventory requires the full CI path.

Running Specific Tests

./scripts/repo-cargo test test_name -- --exact

./scripts/repo-cargo test -p elephant-api
./scripts/repo-cargo test -p elephant-pipeline
./scripts/repo-cargo test -p elephant-storage

./scripts/repo-cargo test -p elephant-scenarios -- --nocapture

Makefile Targets

The Makefile is the single source of truth for local hooks, CI, and release commands:

Target	Description
`make test`	Fast workspace test lane
`make test-unit`	Workspace unit/binary lane
`make test-all`	Full all-features workspace test lane
`make hook-test`	Pre-push unit lane
`make hook-test-all`	Pre-push full lane
`make test-scenarios`	Scenario tests only
`make hook-lint`	Run clippy
`make fmt`	Fix formatting
`make verify-surface`	Verify active surface and legacy drift
`make verify-version-parity`	Verify crate/workspace version alignment
`make ci`	Full CI pipeline
`make ci-smoke`	Faster smoke CI lane
`make release-preflight`	Release validation lane
`make publish-dry-run`	Cargo package dry-run for publishable crates
`make install-hooks`	Install pre-commit hooks

Pre-Commit Hooks

Install hooks with:

pip install pre-commit
pre-commit install
pre-commit install --hook-type pre-push

Or use the Makefile shortcut: make install-hooks.

On Commit

The commit-stage hook is intentionally small:

Hook	Description
`cargo-fmt-fix`	Auto-format Rust files via `./scripts/repo-cargo fmt --all`

On Push

Hook	Description
`gitleaks`	Secret detection
file hygiene hooks	whitespace, EOF, YAML/TOML, merge conflict, key detection
`cargo-fmt`	`make fmt-check`
`cargo-clippy`	`make hook-lint`
`verify-surface`	`make verify-surface`
`cargo-test`	workspace unit gate via `scripts/pre-push-unit.sh`

Heavy all-features tests, docs, E2E, WASM, and audit run in CI/release lanes rather than every local pre-push.

Testing Tiers

Meerkat Recall uses four testing tiers with increasing scope:

Unit tests

Domain and repository behavior tested using #[cfg(test)] modules within each crate. Run the unit/binary lane with make test-unit.No external dependencies required. Tests run entirely in-process.

Integration tests

Storage tests use the in-process kv-mem backend. Some expensive and end-to-end tests are marked #[ignore]; make test-e2e selects deterministic ignored checks. ./scripts/repo-cargo integration runs all ignored tests and may include provider-backed scenarios.Each test creates a unique namespace/database pair for isolation:

let config = StorageConfig {
    url: "mem://".to_string(),
    namespace: format!("ns_{}", uuid),
    database: format!("db_{}", uuid),
    ..StorageConfig::default()
};

E2E Scenarios

System-level tests verifying cross-cutting invariants. See the Scenarios page for the full test map.Run with:

./scripts/repo-cargo test -p elephant-scenarios -- --nocapture

Prompt calibration

Golden-case suites in tests/calibration (package elephant-calibration) score the LLM prompt entrypoints — truth-maintenance adjudication and entity disambiguation — per field against labeled expected verdicts, writing results.json for prompt-version diffing. The real-LLM runners are #[ignore]d and skip cleanly when ANTHROPIC_API_KEY is unset. See tests/calibration/README.md.

ANTHROPIC_API_KEY=... ./scripts/repo-cargo test -p elephant-calibration -- --ignored --nocapture

Workspace Lints

The workspace enforces strict linting rules:

Lint	Level	Description
`unsafe_code`	`forbid`	Unsafe code is completely prohibited
`unwrap_used`	`warn`	Prefer error handling over `.unwrap()`
`expect_used`	`warn`	Prefer error handling over `.expect()`
`panic`	`warn`	Avoid explicit panics
`todo`	`warn`	Avoid the `todo!()` macro in production code
Clippy `pedantic`	enabled	Pedantic lints for code quality
Clippy `nursery`	enabled	Experimental lints for early issue detection

All clippy warnings are treated as errors (-D warnings) in CI and pre-push hooks.

Local Development

# Start the development database and apply the loopback-only dev environment
docker compose -f docker-compose.dev.yml up -d
source .envrc
./scripts/repo-cargo run -p meerkat-recall-bin -- migrate

# Start Meerkat Recall API / MCP runtime
./scripts/repo-cargo run -p meerkat-recall-bin -- serve

This example sets SKIP_AUTH=true and binds to loopback for local development. Use MCP get_runtime_status (or GET /v1/runtime/status) to inspect readiness. Without storage environment variables or persisted configuration, Meerkat Recall starts in bootstrap mode. Authenticated startup still requires a valid JWT_SECRET. The preferred local project config lives at .elephant/elephant_config.toml. For an isolated embedded demo:

SKIP_AUTH=true ELEPHANT_HOST=127.0.0.1 ELEPHANT_DB=elephant-db \
  ./scripts/repo-cargo run -p meerkat-recall-bin -- serve

An optional bootstrap snapshot can be selected with ELEPHANT_DB_SNAPSHOT_LOAD=/tmp/elephant.json. The .cargo/config.toml sets default environment variables for development:

RUST_BACKTRACE=1 for full backtraces
RUST_LOG=info,elephant=debug for useful log output

Diagnose bulk ingestion

Before a bulk run, inspect the selected space’s effective daily budget and reservation-aware headroom through GET /api/v1/budget; see operator budget setup. For oversized inputs, preserve source bytes and choose a new approved text-full-v2 profile with the guarded reindex sequence. Do not relabel existing text-v1 vectors or silently truncate input. Authorized work-run metrics retain custom.failure_diagnostic with fixed kind, stage, code and action fields. Worker warnings expose the matching failure_kind, failure_stage, failure_code and failure_action. Use these typed causes to distinguish model input/output limits, provider quota, MCP transport, extraction budget and schema failures. A new diagnostic cannot recover a cause already absent from a historical receipt. Keep raw source text, provider error bodies and credentials out of shared reports.

Operations

Contributing

Key Dependencies

Repository Structure

Build Commands

CI scope and resource tests

Running Specific Tests

Makefile Targets

Pre-Commit Hooks

On Commit

On Push

Testing Tiers

Workspace Lints

Local Development

Diagnose bulk ingestion

​Key Dependencies

​Repository Structure

​Build Commands

​CI scope and resource tests

​Running Specific Tests

​Makefile Targets

​Pre-Commit Hooks

​On Commit

​On Push

​Testing Tiers

​Workspace Lints

​Local Development

​Diagnose bulk ingestion

Key Dependencies

Repository Structure

Build Commands

CI scope and resource tests

Running Specific Tests

Makefile Targets

Pre-Commit Hooks

On Commit

On Push

Testing Tiers

Workspace Lints

Local Development

Diagnose bulk ingestion