> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rkat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Observability

> Logging configuration, health endpoints, metrics, and troubleshooting runbook

## Logging

Meerkat Recall uses structured logging via the `tracing` crate. Control log verbosity with the `RUST_LOG` environment variable.

### RUST\_LOG Configuration

| Value | Use Case | Description |
| - | - | - |
| `info` | Production | Default level. Logs request lifecycle, worker progress, errors. |
| `info,elephant=debug` | Field testing | Recommended. Adds debug output for Meerkat Recall crates only. |
| `debug` | Deep troubleshooting | All debug output including third-party crates. Very verbose. |
| `warn` | Minimal | Only warnings and errors. |
| `info,elephant_pipeline=debug` | Pipeline debugging | Debug output for pipeline workers only. |
| `info,elephant_storage=debug` | Storage debugging | Debug output for database queries only. |

Set `RUST_LOG` in the `elephant` service environment in `docker-compose.yml`, then recreate that service with `docker compose up -d elephant`. For a directly launched binary, set it in the process environment.

### Structured Log Fields

Logs carry timestamps and tracing targets. Individual events may include request, work item, document, or span identifiers. Use the fields actually present to correlate a failure; a request ID is not guaranteed on every event.

### Viewing Logs

<CodeGroup>
  ```bash Docker Compose theme={null}
  # All services
  docker compose logs -f

  # Server only
  docker compose logs -f elephant

  # SurrealDB only
  docker compose logs -f surrealdb
  ```

  ```bash Local development theme={null}
  # Capture the process logs using your terminal or service supervisor
  RUST_LOG=info,elephant=debug ./scripts/repo-cargo run -p meerkat-recall-bin -- serve
  ```
</CodeGroup>

## Health Endpoints

| Endpoint | Service | Description |
| - | - | - |
| `GET /v1/health` | meerkat-recall serve | Returns API server health with backend + storage status |
| `GET localhost:8000/health` | surrealdb | SurrealDB native health check |
| `GET /v1/runtime/status` | meerkat-recall serve | Backend capability and semantic readiness flags |
| MCP `get_health` | meerkat-recall serve | Programmatic health check via MCP tool |

The API health endpoint returns:

```json theme={null}
{
  "status": "healthy",
  "backend": "surreal-db",
  "storage": "connected",
  "surrealdb": "connected",
  "embedding_provider": "healthy",
  "circuit_breakers": {
    "storage": "closed",
    "surrealdb": "closed",
    "embeddings": "closed"
  }
}
```

This health response is illustrative; it is not evidence of completed extraction or semantic readiness. Inspect `/v1/runtime/status`, required readiness flags, and representative work completion before accepting a deployment.

`storage` is the real backend-agnostic field. `surrealdb` remains as a compatibility alias for older clients.

Docker Compose uses these for container health checks:

* **meerkat-recall serve**: `curl -f http://localhost:3000/v1/health` every 30s
* **surrealdb**: `/surreal is-ready` every 10s

## Key Metrics to Monitor

These are operational signals to derive from your logs, proxy, runtime status, and queue inspection. This table does not imply a built-in metrics exporter. Thresholds are examples to tune for your workload.

### API Metrics

| Metric | What to Watch |
| - | - |
| Request count | Baseline throughput; sudden drops indicate problems |
| Error rate (4xx, 5xx) | Spike above 5% over 5 minutes warrants investigation |
| Latency (p50/p95) | p95 above 2s may indicate storage contention or worker backlog |
| Active connections | Sustained high counts can exhaust connection pool |

### Pipeline Metrics

| Metric | What to Watch |
| - | - |
| Embed worker throughput | Embeddings generated per minute |
| Work queue depth | Pending/runnable items; sustained growth indicates backlog |
| Work item failure rate | Consecutive failures above 10 suggest a systemic issue |
| Extraction latency | Time from doc ingest to assertions extracted |
| Truth maintenance events | Conflict detection and resolution counts |
| Retention cleanup | Records cleaned per cycle |

### SurrealDB Metrics

| Metric | What to Watch |
| - | - |
| Connection errors | More than 3 consecutive failures is critical |
| Query latency | Slow queries indicate missing indexes or contention |
| Disk usage | Monitor `surrealdb_data` volume growth |

## Troubleshooting Runbook

| Symptom | Likely Cause | Resolution |
| - | - | - |
| `/v1/health` returns non-200 | Storage unreachable or API crash | Check the `backend` and `storage` fields in the response, then verify the configured backend is healthy |
| `401 Unauthorized` on API calls | Missing or invalid JWT token | Verify `JWT_SECRET` matches token signer, or set `SKIP_AUTH=true` for development |
| Embeddings not generated | Missing API key or wrong provider | Check `OPENAI_API_KEY` is set and `PIPELINE_EMBED_PROVIDER` is correct |
| Pipeline not processing work items | Workers disabled or SurrealDB connection failed | Verify `PIPELINE_*_ENABLED` flags are `true`; check pipeline logs for connection errors |
| High memory usage | Large batch sizes or concurrent limit too high | Reduce `PIPELINE_EMBED_BATCH_SIZE` and `PIPELINE_MAX_CONCURRENT` |
| Work items stuck in `running` | A long-running handler, a crashed worker, or queue contention | Inspect the worker logs and lease timestamp. Expired leases become recoverable after a 60-second grace period and a scheduler pass. `retry_work_item` accepts only `failed` items |
| Extraction failures | Configured model unavailable or provider rate limited | Verify the extraction model and its provider credentials; inspect provider errors |
| Entity disambiguation failures during artifact commit | LLM timeout or model unavailable | Inspect the promotion error, `PIPELINE_PROMOTION_DISAMBIGUATION_MODEL`, and Anthropic credentials; the `PIPELINE_RESOLUTION_*` settings belong to the compatibility resolution path |
| SurrealDB "read or write conflict" | Transaction contention under high concurrency | Transient; the pipeline retries automatically up to `PIPELINE_WORK_TRANSIENT_RETRY_MAX_ATTEMPTS` times |
| Disk space exhaustion | SurrealDB data volume full | Increase capacity or use a reviewed data-retention process. Queue archival reduces active scans but retains audit payloads; it does not guarantee disk reclamation |
| Pipeline stalled (no progress in 15 min while work is queued) | Database contention, a growing queue, or failed workers | Inspect runtime flags and work-item status, then follow the [capacity guide](https://github.com/lukacf/meerkat-recall/blob/d71722482183a802fcaeaaf41cb04ef7d16f4b85/docs/work-queue-capacity.md) before restarting the managed service |

For an accepted document that is not searchable yet, inspect its
[pipeline status](/recall/api/documents#get-document-pipeline-status). A healthy API,
a successful ingest request, and a `done` work item each answer a different
question. Check the relevant downstream extraction and embedding stages before
treating a document as ready for your application.

## Alerting Recommendations

| Alert | Condition | Severity |
| - | - | - |
| API unhealthy | `/v1/health` returns non-200 for 60s | Critical |
| Error rate spike | HTTP 5xx rate > 5% over 5 minutes | High |
| Pipeline stalled | Work remains queued and no work items complete in 15 minutes | Medium |
| SurrealDB unreachable | Connection failures > 3 consecutive | Critical |
| Embedding failures | Consecutive embed failures > 10 | High |
| Work queue backlog | Pending items > 1000 sustained for 30 min | Medium |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.