Extraction

The extraction system transforms document nodes into structured assertions (subject-predicate-object triples). It uses a skill resolution cascade to determine what to extract and an LLM-based extractor to perform the actual extraction.

Skill Resolution Cascade

When a document enters the extraction pipeline, the system resolves two pieces of guidance: a purpose (what the space cares about) and a skill (domain-specific extraction instructions). Each is resolved by checking sources in priority order — the first non-empty value wins.

Purpose Resolution

Priority	Source	Config Key
1	Space config	`space_config.purpose`
2	Document metadata	`doc.metadata.extraction_purpose` or `doc.metadata.purpose`
3	Source metadata	`source.metadata.extraction_purpose` or `source.metadata.purpose`
4	Environment default	`PIPELINE_DEFAULT_EXTRACTION_PURPOSE`

Skill Resolution

Priority	Source	Config Key
1	Space config	`space_config.skill`
2	Document metadata	First populated `extraction_skill`, `extraction_skill_id`, `skill`, or `skill_id`
3	Source metadata	Same skill fields as document metadata
4	Document domain mapping	First populated `extraction_domain`, `domain`, `genre`, or `content_domain`, mapped to a built-in skill
5	Source domain mapping	Same domain fields as document metadata
6	Source type env map	`PIPELINE_SKILL_BY_SOURCE_TYPE_JSON` (JSON map, e.g. `{"email":"extraction/custom-email","*":"extraction/technical-docs"}`)
7	Source type default	Built-in default skill for the source type
8	Environment default	`PIPELINE_DEFAULT_EXTRACTION_SKILL`

All current source types default to extraction/technical-docs. In the normal source-backed path, that fallback is reached before PIPELINE_DEFAULT_EXTRACTION_SKILL. To change the default for source-backed ingest, use an explicit space skill or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON (including its "*" fallback); setting only the last environment default does not override the source-type choice. The skill resolution also performs filesystem fuzzy matching: skill IDs are normalized and matched against SKILL.md files in the skills directory (PIPELINE_SKILLS_DIR, default ./skills).

The resolution trace is recorded in the extraction plan artifact and exposed via the get_doc_pipeline_status MCP endpoint for debugging which skill was selected and why.

The most common way to configure extraction is to set purpose and skill on the SpaceConfig - this applies to all documents ingested into that space without requiring per-document metadata.

Built-in skills

Meerkat Recall ships with domain-neutral extraction skills in the skills/extraction/ directory. Each is a SKILL.md file containing markdown guidance that gets injected into the LLM prompt. App-specific domains, including email/chat semantics, should provide their own skill directory and select it through metadata, space config, or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON.

extraction/fiction - novels, stories, narratives

Entity types: person, location, device, group, topicFocus: Character relationships that persist across the narrative - who knows whom, who rules what, family ties, group memberships. Skips momentary actions (Alice picked up a bottle) in favor of durable state (Alice is the protagonist).Key predicates: knows, related_to, parent_of, lives_in, rules_over, possesses, member_of, leads, has_role, has_trait

extraction/technical-docs - architecture docs, specs, runbooks

Entity types: component, platform, concept, data_model, groupFocus: Component relationships, technology choices, architectural patterns, data model relationships, and design decisions. Targets the architecture-diagram level, not implementation code.Key predicates: depends_on, part_of, communicates_with, uses, implements, stores_in, chosen_over, replaces

You can create custom skills by adding a new SKILL.md file under skills/your-domain/. The skill ID is the directory path relative to the skills root (e.g., your-domain or extraction/your-domain).

Extraction Context

The extractor receives an ExtractionContext containing everything it needs:

Field	Description
`node_text`	The text content of the document node being extracted
`purpose`	Space purpose from the skill resolution cascade
`skill_body`	Domain extraction instructions loaded from `SKILL.md`
`known_entities`	Entities already identified from earlier chunks of the same document
`budget`	Token and wall-time limits for this extraction
`doc_id`	Document ID for tool-based text access (agentic extractor only)

Extractors

Agentic Extractor (Meerkat-based)

The primary extractor, enabled via the meerkat-extraction feature flag. It uses Meerkat - a modular LLM agent harness - to orchestrate extraction as a structured conversation with tool access.

Model: configurable via PIPELINE_AGENTIC_MODEL (binary default: gpt-5.6-sol; Docker Compose selects claude-opus-4-6). Supply credentials for the selected provider.
Structured output: Meerkat receives a strict assertion schema and a bounded schema-repair retry count. Meerkat Recall parses and validates the result before staging candidates. Invalid output or an incomplete run can still fail the work item; a schema is not a guarantee of factual correctness.
Tool access: the normal agentic path provides ReadTextRangeDispatcher for text-span checks and ElephantToolBridge for querying existing entities and predicates. Large text spans use a direct structured extraction path without those tools. An empty result from the agentic path can also trigger one direct structured retry.
Budget enforcement: the extractor uses the lower token and duration limits from its configuration and extraction context (defaults: 500,000 tokens and 600 seconds). The worker’s lifecycle timeout is a separate bound; see work queue configuration.
Progress: the extraction call returns a completed result to the worker. Inspect document pipeline status and work-run logs for stage progress; this is not a public stream of partially committed assertions.

The agentic extractor can resolve ambiguities by querying the knowledge graph during extraction, reducing duplicate entity creation.

Non-Active Fallbacks

Meerkat Recall no longer uses a rule-based semantic fallback in the active runtime. If the real extractor is unavailable, semantic ingest is hidden and extraction work fails loudly with an explicit runtime/work-item error. Any non-LLM extraction helpers that remain in the codebase are test-only or legacy compatibility scaffolding, not part of the supported product path.

RawAssertion Output

Each extractor produces a list of RawAssertion records - pre-resolution claims with mention strings. This excerpt shows the main claim fields; the type also carries quote, entity-type, and event-lineage hints:

pub struct RawAssertion {
    pub subject_text: String,          // "Alice"
    pub predicate_text: String,        // "works at"
    pub object_text: Option<String>,   // "Acme Corp" (entity-to-entity)
    pub object_literal: Option<Literal>, // or a typed literal value
    pub evidence_span: EvidenceSpan,   // byte offsets into source text
    pub confidence: f64,               // 0.0 to 1.0
    pub source_event_time: Option<DateTime<Utc>>,
    pub subject_entity_id: Option<String>,  // existing entity if found
    pub object_entity_id: Option<String>,
    pub significance: Option<String>,  // "skip", "low", "medium", "high"
}

Assertions with significance: "skip" are recorded in the evidence layer but do not produce rels or attributes. This allows the extractor to note observations without polluting the knowledge graph.

Artifact Commit Boundary

After extraction, Meerkat Recall does not write graph rows directly from raw assertions. Instead:

extract_assertions stages knowledge_candidates
validate_artifacts validates those candidates
commit_artifacts performs canonical writes

That commit phase is what creates assertions, relationships, attributes, and events:

Resolve Subject Entity

Look up by subject_entity_id (if the LLM found a match) or by identity key (lowercased display name). Identity-key reuse is type-guarded: all matches are fetched and the incoming type (normalized first) must be compatible with the candidate’s — topic is the unknown bucket and acts as a wildcard, upgrading to the specific incoming type on reuse. Ambiguous or type-incompatible matches route to an LLM disambiguator using the current space’s approved promotion_disambiguation binding; without one, incompatible types create a new entity rather than risking a false merge. Create if nothing matches.

Resolve Object Entity

Same process for entity-to-entity assertions. Skipped for literal-valued assertions.

Create Assertion Record

Always created in the evidence layer with the full evidence span and confidence score.

Create Rel or Attribute

If significance is not “skip”: creates a rel (entity-to-entity) or attribute (entity-to-literal) record.

Write Outbox Event

A batched outbox event is written with the created/updated record references. Truth maintenance picks this up asynchronously.

Diagnose an extraction failure

Use get_doc_pipeline_status on /mcp/full with the returned document ID, then inspect failed work items in the same space. Separate these cases before retrying:

Observation	Check next
No extraction was triggered	Requested ingest mode and runtime semantic readiness
Plan selects an unexpected skill	Resolution trace, space configuration, and the worker’s skills directory
Provider or model error	Credentials and model selection in the worker process
Budget or timeout error	Selected node size, extractor limits, and worker lifecycle timeout
Extraction completes with few claims	Source text, selected nodes, and skill instructions; claim counts are model-dependent
Candidates fail validation	Recorded validation errors and evidence spans; retrying the same invalid payload alone does not correct them

See ingesting data for a source-to-graph procedure and work queue recovery for retry semantics.

Pipeline

Architecture

Skill Resolution Cascade

Purpose Resolution

Skill Resolution

Built-in skills

Extraction Context

Extractors

Agentic Extractor (Meerkat-based)

Non-Active Fallbacks

RawAssertion Output

Artifact Commit Boundary

Diagnose an extraction failure

​Skill Resolution Cascade

​Purpose Resolution

​Skill Resolution

​Built-in skills

​Extraction Context

​Extractors

​Agentic Extractor (Meerkat-based)

​Non-Active Fallbacks

​RawAssertion Output

​Artifact Commit Boundary

​Diagnose an extraction failure

Skill Resolution Cascade

Purpose Resolution

Skill Resolution

Built-in skills

Extraction Context

Extractors

Agentic Extractor (Meerkat-based)

Non-Active Fallbacks

RawAssertion Output

Artifact Commit Boundary

Diagnose an extraction failure