Observability

Logging

Meerkat Recall uses structured logging via the tracing crate. Control log verbosity with the RUST_LOG environment variable.

RUST_LOG Configuration

Value	Use Case	Description
`info`	Production	Default level. Logs request lifecycle, worker progress, errors.
`info,elephant=debug`	Field testing	Recommended. Adds debug output for Meerkat Recall crates only.
`debug`	Deep troubleshooting	All debug output including third-party crates. Very verbose.
`warn`	Minimal	Only warnings and errors.
`info,elephant_pipeline=debug`	Pipeline debugging	Debug output for pipeline workers only.
`info,elephant_storage=debug`	Storage debugging	Debug output for database queries only.

Set RUST_LOG in the elephant service environment in docker-compose.yml, then recreate that service with docker compose up -d elephant. For a directly launched binary, set it in the process environment.

Structured Log Fields

Logs carry timestamps and tracing targets. Individual events may include request, work item, document, or span identifiers. Use the fields actually present to correlate a failure; a request ID is not guaranteed on every event.

Viewing Logs

# All services
docker compose logs -f

# Server only
docker compose logs -f elephant

# SurrealDB only
docker compose logs -f surrealdb

# Capture the process logs using your terminal or service supervisor
RUST_LOG=info,elephant=debug ./scripts/repo-cargo run -p meerkat-recall-bin -- serve

Health Endpoints

Endpoint	Service	Description
`GET /v1/health`	meerkat-recall serve	Returns API server health with backend + storage status
`GET localhost:8000/health`	surrealdb	SurrealDB native health check
`GET /v1/runtime/status`	meerkat-recall serve	Backend capability and semantic readiness flags
MCP `get_health`	meerkat-recall serve	Programmatic health check via MCP tool

The API health endpoint returns:

{
  "status": "healthy",
  "backend": "surreal-db",
  "storage": "connected",
  "surrealdb": "connected",
  "embedding_provider": "healthy",
  "circuit_breakers": {
    "storage": "closed",
    "surrealdb": "closed",
    "embeddings": "closed"
  }
}

This health response is illustrative; it is not evidence of completed extraction or semantic readiness. Inspect /v1/runtime/status, required readiness flags, and representative work completion before accepting a deployment. storage is the real backend-agnostic field. surrealdb remains as a compatibility alias for older clients. Docker Compose uses these for container health checks:

meerkat-recall serve: curl -f http://localhost:3000/v1/health every 30s
surrealdb: /surreal is-ready every 10s

Key Metrics to Monitor

These are operational signals to derive from your logs, proxy, runtime status, and queue inspection. This table does not imply a built-in metrics exporter. Thresholds are examples to tune for your workload.

API Metrics

Metric	What to Watch
Request count	Baseline throughput; sudden drops indicate problems
Error rate (4xx, 5xx)	Spike above 5% over 5 minutes warrants investigation
Latency (p50/p95)	p95 above 2s may indicate storage contention or worker backlog
Active connections	Sustained high counts can exhaust connection pool

Pipeline Metrics

Metric	What to Watch
Embed worker throughput	Embeddings generated per minute
Work queue depth	Pending/runnable items; sustained growth indicates backlog
Work item failure rate	Consecutive failures above 10 suggest a systemic issue
Extraction latency	Time from doc ingest to assertions extracted
Truth maintenance events	Conflict detection and resolution counts
Retention cleanup	Records cleaned per cycle

SurrealDB Metrics

Metric	What to Watch
Connection errors	More than 3 consecutive failures is critical
Query latency	Slow queries indicate missing indexes or contention
Disk usage	Monitor `surrealdb_data` volume growth

Troubleshooting Runbook

Symptom	Likely Cause	Resolution
`/v1/health` returns non-200	Storage unreachable or API crash	Check the `backend` and `storage` fields in the response, then verify the configured backend is healthy
`401 Unauthorized` on API calls	Missing or invalid JWT token	Verify `JWT_SECRET` matches token signer, or set `SKIP_AUTH=true` for development
Embeddings not generated	Missing API key or wrong provider	Check `OPENAI_API_KEY` is set and `PIPELINE_EMBED_PROVIDER` is correct
Pipeline not processing work items	Workers disabled or SurrealDB connection failed	Verify `PIPELINE_*_ENABLED` flags are `true`; check pipeline logs for connection errors
High memory usage	Large batch sizes or concurrent limit too high	Reduce `PIPELINE_EMBED_BATCH_SIZE` and `PIPELINE_MAX_CONCURRENT`
Work items stuck in `running`	A long-running handler, a crashed worker, or queue contention	Inspect the worker logs and lease timestamp. Expired leases become recoverable after a 60-second grace period and a scheduler pass. `retry_work_item` accepts only `failed` items
Extraction failures	Configured model unavailable or provider rate limited	Verify the extraction model and its provider credentials; inspect provider errors
Entity disambiguation failures during artifact commit	LLM timeout or model unavailable	Inspect the promotion error, `PIPELINE_PROMOTION_DISAMBIGUATION_MODEL`, and Anthropic credentials; the `PIPELINE_RESOLUTION_*` settings belong to the compatibility resolution path
SurrealDB “read or write conflict”	Transaction contention under high concurrency	Transient; the pipeline retries automatically up to `PIPELINE_WORK_TRANSIENT_RETRY_MAX_ATTEMPTS` times
Disk space exhaustion	SurrealDB data volume full	Increase capacity or use a reviewed data-retention process. Queue archival reduces active scans but retains audit payloads; it does not guarantee disk reclamation
Pipeline stalled (no progress in 15 min while work is queued)	Database contention, a growing queue, or failed workers	Inspect runtime flags and work-item status, then follow the capacity guide before restarting the managed service

For an accepted document that is not searchable yet, inspect its pipeline status. A healthy API, a successful ingest request, and a done work item each answer a different question. Check the relevant downstream extraction and embedding stages before treating a document as ready for your application.

Alerting Recommendations

Alert	Condition	Severity
API unhealthy	`/v1/health` returns non-200 for 60s	Critical
Error rate spike	HTTP 5xx rate > 5% over 5 minutes	High
Pipeline stalled	Work remains queued and no work items complete in 15 minutes	Medium
SurrealDB unreachable	Connection failures > 3 consecutive	Critical
Embedding failures	Consecutive embed failures > 10	High
Work queue backlog	Pending items > 1000 sustained for 30 min	Medium

Operations

Contributing

Logging

RUST_LOG Configuration

Structured Log Fields

Viewing Logs

Health Endpoints

Key Metrics to Monitor

API Metrics

Pipeline Metrics

SurrealDB Metrics

Troubleshooting Runbook

Alerting Recommendations

​Logging

​RUST_LOG Configuration

​Structured Log Fields

​Viewing Logs

​Health Endpoints

​Key Metrics to Monitor

​API Metrics

​Pipeline Metrics

​SurrealDB Metrics

​Troubleshooting Runbook

​Alerting Recommendations

Logging

RUST_LOG Configuration

Structured Log Fields

Viewing Logs

Health Endpoints

Key Metrics to Monitor

API Metrics

Pipeline Metrics

SurrealDB Metrics

Troubleshooting Runbook

Alerting Recommendations