> ## Documentation Index
> Fetch the complete documentation index at: https://docs.rkat.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Extraction

> How Meerkat Recall selects extraction skills and transforms documents into structured assertions.

The extraction system transforms document nodes into structured assertions (subject-predicate-object triples). It uses a skill resolution cascade to determine **what** to extract and an LLM-based extractor to perform the actual extraction.

## Skill Resolution Cascade

When a document enters the extraction pipeline, the system resolves two pieces of guidance: a **purpose** (what the space cares about) and a **skill** (domain-specific extraction instructions). Each is resolved by checking sources in priority order -- the first non-empty value wins.

### Purpose Resolution

| Priority | Source | Config Key |
| - | - | - |
| 1 | Space config | `space_config.purpose` |
| 2 | Document metadata | `doc.metadata.extraction_purpose` or `doc.metadata.purpose` |
| 3 | Source metadata | `source.metadata.extraction_purpose` or `source.metadata.purpose` |
| 4 | Environment default | `PIPELINE_DEFAULT_EXTRACTION_PURPOSE` |

### Skill Resolution

| Priority | Source | Config Key |
| - | - | - |
| 1 | Space config | `space_config.skill` |
| 2 | Document metadata | First populated `extraction_skill`, `extraction_skill_id`, `skill`, or `skill_id` |
| 3 | Source metadata | Same skill fields as document metadata |
| 4 | Document domain mapping | First populated `extraction_domain`, `domain`, `genre`, or `content_domain`, mapped to a built-in skill |
| 5 | Source domain mapping | Same domain fields as document metadata |
| 6 | Source type env map | `PIPELINE_SKILL_BY_SOURCE_TYPE_JSON` (JSON map, e.g. `{"email":"extraction/custom-email","*":"extraction/technical-docs"}`) |
| 7 | Source type default | Built-in default skill for the source type |
| 8 | Environment default | `PIPELINE_DEFAULT_EXTRACTION_SKILL` |

All current source types default to `extraction/technical-docs`. In the normal
source-backed path, that fallback is reached before
`PIPELINE_DEFAULT_EXTRACTION_SKILL`. To change the default for source-backed
ingest, use an explicit space skill or `PIPELINE_SKILL_BY_SOURCE_TYPE_JSON`
(including its `"*"` fallback); setting only the last environment default does
not override the source-type choice.

The skill resolution also performs **filesystem fuzzy matching**: skill IDs are normalized and matched against `SKILL.md` files in the skills directory (`PIPELINE_SKILLS_DIR`, default `./skills`).

<Note>
  The resolution trace is recorded in the extraction plan artifact and exposed via the `get_doc_pipeline_status` MCP endpoint for debugging which skill was selected and why.
</Note>

The most common way to configure extraction is to set `purpose` and `skill` on the [SpaceConfig](/recall/concepts/spaces#extraction-skill-routing) - this applies to all documents ingested into that space without requiring per-document metadata.

### Built-in skills

Meerkat Recall ships with domain-neutral extraction skills in the `skills/extraction/` directory. Each is a `SKILL.md` file containing markdown guidance that gets injected into the LLM prompt. App-specific domains, including email/chat semantics, should provide their own skill directory and select it through metadata, space config, or `PIPELINE_SKILL_BY_SOURCE_TYPE_JSON`.

<Accordion title="extraction/fiction - novels, stories, narratives">
  **Entity types:** person, location, device, group, topic

  **Focus:** Character relationships that persist across the narrative - who knows whom, who rules what, family ties, group memberships. Skips momentary actions (Alice picked up a bottle) in favor of durable state (Alice is the protagonist).

  **Key predicates:** `knows`, `related_to`, `parent_of`, `lives_in`, `rules_over`, `possesses`, `member_of`, `leads`, `has_role`, `has_trait`
</Accordion>

<Accordion title="extraction/technical-docs - architecture docs, specs, runbooks">
  **Entity types:** component, platform, concept, data\_model, group

  **Focus:** Component relationships, technology choices, architectural patterns, data model relationships, and design decisions. Targets the architecture-diagram level, not implementation code.

  **Key predicates:** `depends_on`, `part_of`, `communicates_with`, `uses`, `implements`, `stores_in`, `chosen_over`, `replaces`
</Accordion>

You can create custom skills by adding a new `SKILL.md` file under `skills/your-domain/`. The skill ID is the directory path relative to the skills root (e.g., `your-domain` or `extraction/your-domain`).

## Extraction Context

The extractor receives an `ExtractionContext` containing everything it needs:

| Field | Description |
| - | - |
| `node_text` | The text content of the document node being extracted |
| `purpose` | Space purpose from the skill resolution cascade |
| `skill_body` | Domain extraction instructions loaded from `SKILL.md` |
| `known_entities` | Entities already identified from earlier chunks of the same document |
| `budget` | Token and wall-time limits for this extraction |
| `doc_id` | Document ID for tool-based text access (agentic extractor only) |

## Extractors

### Agentic Extractor (Meerkat-based)

The primary extractor, enabled via the `meerkat-extraction` feature flag. It uses [Meerkat](https://github.com/lukacf/meerkat) - a modular LLM agent harness - to orchestrate extraction as a structured conversation with tool access.

* **Model:** configurable via `PIPELINE_AGENTIC_MODEL` (binary default: `gpt-5.6-sol`; Docker Compose selects `claude-opus-4-6`). Supply credentials for the selected provider.
* **Structured output:** Meerkat receives a strict assertion schema and a bounded schema-repair retry count. Meerkat Recall parses and validates the result before staging candidates. Invalid output or an incomplete run can still fail the work item; a schema is not a guarantee of factual correctness.
* **Tool access:** the normal agentic path provides `ReadTextRangeDispatcher` for text-span checks and `ElephantToolBridge` for querying existing entities and predicates. Large text spans use a direct structured extraction path without those tools. An empty result from the agentic path can also trigger one direct structured retry.
* **Budget enforcement:** the extractor uses the lower token and duration limits from its configuration and extraction context (defaults: 500,000 tokens and 600 seconds). The worker's lifecycle timeout is a separate bound; see [work queue configuration](/recall/operations/deployment#pipeline-work-queue).
* **Progress:** the extraction call returns a completed result to the worker. Inspect document pipeline status and work-run logs for stage progress; this is not a public stream of partially committed assertions.

The agentic extractor can resolve ambiguities by querying the knowledge graph during extraction, reducing duplicate entity creation.

### Non-Active Fallbacks

Meerkat Recall no longer uses a rule-based semantic fallback in the active runtime. If the real extractor is unavailable, semantic ingest is hidden and extraction work fails loudly with an explicit runtime/work-item error.

Any non-LLM extraction helpers that remain in the codebase are test-only or legacy compatibility scaffolding, not part of the supported product path.

## RawAssertion Output

Each extractor produces a list of `RawAssertion` records - pre-resolution claims with mention strings. This excerpt shows the main claim fields; the type also carries quote, entity-type, and event-lineage hints:

```rust theme={null}
pub struct RawAssertion {
    pub subject_text: String,          // "Alice"
    pub predicate_text: String,        // "works at"
    pub object_text: Option<String>,   // "Acme Corp" (entity-to-entity)
    pub object_literal: Option<Literal>, // or a typed literal value
    pub evidence_span: EvidenceSpan,   // byte offsets into source text
    pub confidence: f64,               // 0.0 to 1.0
    pub source_event_time: Option<DateTime<Utc>>,
    pub subject_entity_id: Option<String>,  // existing entity if found
    pub object_entity_id: Option<String>,
    pub significance: Option<String>,  // "skip", "low", "medium", "high"
}
```

<Warning>
  Assertions with `significance: "skip"` are recorded in the evidence layer but do not produce rels or attributes. This allows the extractor to note observations without polluting the knowledge graph.
</Warning>

## Artifact Commit Boundary

After extraction, Meerkat Recall does not write graph rows directly from raw assertions. Instead:

1. `extract_assertions` stages `knowledge_candidates`
2. `validate_artifacts` validates those candidates
3. `commit_artifacts` performs canonical writes

That commit phase is what creates assertions, relationships, attributes, and events:

<Steps>
  <Step title="Resolve Subject Entity">
    Look up by `subject_entity_id` (if the LLM found a match) or by identity key (lowercased display name). Identity-key reuse is type-guarded: all matches are fetched and the incoming type (normalized first) must be compatible with the candidate's -- `topic` is the unknown bucket and acts as a wildcard, upgrading to the specific incoming type on reuse. Ambiguous or type-incompatible matches route to an LLM disambiguator using the current space's approved `promotion_disambiguation` binding; without one, incompatible types create a new entity rather than risking a false merge. Create if nothing matches.
  </Step>

  <Step title="Resolve Object Entity">
    Same process for entity-to-entity assertions. Skipped for literal-valued assertions.
  </Step>

  <Step title="Create Assertion Record">
    Always created in the evidence layer with the full evidence span and confidence score.
  </Step>

  <Step title="Create Rel or Attribute">
    If significance is not "skip": creates a `rel` (entity-to-entity) or `attribute` (entity-to-literal) record.
  </Step>

  <Step title="Write Outbox Event">
    A batched outbox event is written with the created/updated record references. Truth maintenance picks this up asynchronously.
  </Step>
</Steps>

## Diagnose an extraction failure

Use `get_doc_pipeline_status` on `/mcp/full` with the returned document ID, then
inspect failed work items in the same space. Separate these cases before retrying:

| Observation | Check next |
| - | - |
| No extraction was triggered | Requested ingest mode and runtime semantic readiness |
| Plan selects an unexpected skill | Resolution trace, space configuration, and the worker's skills directory |
| Provider or model error | Credentials and model selection in the worker process |
| Budget or timeout error | Selected node size, extractor limits, and worker lifecycle timeout |
| Extraction completes with few claims | Source text, selected nodes, and skill instructions; claim counts are model-dependent |
| Candidates fail validation | Recorded validation errors and evidence spans; retrying the same invalid payload alone does not correct them |

See [ingesting data](/recall/guides/ingesting-data) for a source-to-graph procedure and
[work queue recovery](/recall/pipeline/work-queue#inspect-and-retry-failed-work) for retry semantics.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.