Skip to main content
The extraction system transforms document nodes into structured assertions (subject-predicate-object triples). It uses a skill resolution cascade to determine what to extract and an LLM-based extractor to perform the actual extraction.

Skill Resolution Cascade

When a document enters the extraction pipeline, the system resolves two pieces of guidance: a purpose (what the space cares about) and a skill (domain-specific extraction instructions). Each is resolved by checking sources in priority order — the first non-empty value wins.

Purpose Resolution

Skill Resolution

All current source types default to extraction/technical-docs. In the normal source-backed path, that fallback is reached before PIPELINE_DEFAULT_EXTRACTION_SKILL. To change the default for source-backed ingest, use an explicit space skill or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON (including its "*" fallback); setting only the last environment default does not override the source-type choice. The skill resolution also performs filesystem fuzzy matching: skill IDs are normalized and matched against SKILL.md files in the skills directory (PIPELINE_SKILLS_DIR, default ./skills).
The resolution trace is recorded in the extraction plan artifact and exposed via the get_doc_pipeline_status MCP endpoint for debugging which skill was selected and why.
The most common way to configure extraction is to set purpose and skill on the SpaceConfig - this applies to all documents ingested into that space without requiring per-document metadata.

Built-in skills

Meerkat Recall ships with domain-neutral extraction skills in the skills/extraction/ directory. Each is a SKILL.md file containing markdown guidance that gets injected into the LLM prompt. App-specific domains, including email/chat semantics, should provide their own skill directory and select it through metadata, space config, or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON.
Entity types: person, location, device, group, topicFocus: Character relationships that persist across the narrative - who knows whom, who rules what, family ties, group memberships. Skips momentary actions (Alice picked up a bottle) in favor of durable state (Alice is the protagonist).Key predicates: knows, related_to, parent_of, lives_in, rules_over, possesses, member_of, leads, has_role, has_trait
Entity types: component, platform, concept, data_model, groupFocus: Component relationships, technology choices, architectural patterns, data model relationships, and design decisions. Targets the architecture-diagram level, not implementation code.Key predicates: depends_on, part_of, communicates_with, uses, implements, stores_in, chosen_over, replaces
You can create custom skills by adding a new SKILL.md file under skills/your-domain/. The skill ID is the directory path relative to the skills root (e.g., your-domain or extraction/your-domain).

Extraction Context

The extractor receives an ExtractionContext containing everything it needs:

Extractors

Agentic Extractor (Meerkat-based)

The model defaults and summary policy on this page describe the current source checkout. Published 0.9.1 binaries retain their original defaults and summary behavior; using the newer defaults and policy requires the newer source. The primary extractor, enabled via the meerkat-extraction feature flag. It uses Meerkat - a modular LLM agent harness - to orchestrate extraction as a structured conversation with tool access.
  • Model: configurable via PIPELINE_AGENTIC_MODEL (binary and Docker Compose default: gpt-6.1-sol). Supply credentials and approved per-space profiles for the selected provider; these defaults do not migrate existing profiles. The pinned Meerkat client uses Responses for GPT-6.1 tool calls.
  • Reasoning: Governed extraction and support verification clear caller reasoning parameters before dispatch. EXTRACTION_REASONING_EFFORT therefore does not select their effective effort, and the current model-profile schema has no reasoning-effort field. GPT-6.1 Sol uses its upstream medium default in this path; it supports low, medium, high, xhigh, and max, but not none or minimal. Selecting an explicit effort for governed extraction or support verification requires a product change.
  • Structured output: Meerkat receives a strict assertion schema and a bounded schema-repair retry count. Meerkat Recall parses and validates the result before staging candidates. Invalid output or an incomplete run can still fail the work item; a schema is not a guarantee of factual correctness.
  • Tool access: the normal agentic path provides ReadTextRangeDispatcher for text-span checks and ElephantToolBridge for querying existing entities and predicates. Large text spans use a direct structured extraction path without those tools. An empty result from the agentic path can also trigger one direct structured retry.
  • Budget enforcement: the extractor uses the lower token and duration limits from its configuration and extraction context (defaults: 500,000 tokens and 600 seconds). The worker’s lifecycle timeout is a separate bound; see work queue configuration.
  • Progress: the extraction call returns a completed result to the worker. Inspect document pipeline status and work-run logs for stage progress; this is not a public stream of partially committed assertions.
The agentic extractor can resolve ambiguities by querying the knowledge graph during extraction, reducing duplicate entity creation.

Non-Active Fallbacks

Meerkat Recall no longer uses a rule-based semantic fallback in the active runtime. If the real extractor is unavailable, semantic ingest is hidden and extraction work fails loudly with an explicit runtime/work-item error. Any non-LLM extraction helpers that remain in the codebase are test-only or legacy compatibility scaffolding, not part of the supported product path.

Summary budgets

The space’s approved summary operation binding selects an immutable model profile. Configure that profile’s summary policy when appending a new profile:
max_output_tokens is the provider’s total output ceiling, including thinking; max_summary_length bounds the retained summary text in Unicode characters. Accepted ranges are 1-65,536 tokens and 1-262,144 characters. Profiles without a summary policy use runtime defaults of 4,096 tokens and 12,000 characters, with the model-specific thinking policy described below. The local token estimate can reduce the retained character ceiling further. These limits do not guarantee that a model will use the full allowance or produce a complete answer. For claude-haiku-5-5, adaptive_low sends adaptive thinking with low effort. The pinned SDK cannot express Haiku’s supported disabled thinking mode, so this policy does not promise zero thinking or a separate thinking-token cap. For tool-free claude-sonnet-5-5 summaries, between_tools_low suppresses up-front thinking and sends low effort. These model-specific policies apply only to the named model on an Anthropic connection. provider_default omits explicit thinking controls. A profile with an explicit summary policy may be bound only to the summary operation; use separate profile IDs for other operations. Only nonempty text with an SDK-reported end-of-turn completion is accepted as a model summary. Empty text, token-limit stops, other known typed incomplete completions, and provider errors use a bounded excerpt of the original text. Fallback and output truncation are logged and counted, without logging source text. Provider approval failures remain errors. The pinned SDK normalizes unknown raw stop reasons, including model_context_window_exceeded, to end-of-turn; those responses can escape incomplete-output detection. Preserving these raw reasons is a tracked SDK follow-up. Inspect these counters under WorkRunMetrics.custom.summary_process_totals.totals on completed work-run metrics: These are cumulative totals for the current process, not counts attributed to one work run. One response can increment multiple counters, such as an empty token-limited response followed by truncation of the local excerpt. The read-only Rust function elephant_pipeline::summarizer::summary_metrics_snapshot() exposes the same totals. Successful transport alone does not establish a useful summary. Append a new profile ID and update the space’s summary binding through the normal authorized processing-configuration replacement. Existing profiles are immutable. During a historical migration, preserve the sealed configuration through final catch-up; apply new-work model choices afterward. Summary policy does not change embeddings or extraction/support-verification reasoning. The summary trigger remains separate from the output allowance: input at or above the configured minimum length (default 1,000 Unicode characters) invokes the summarizer. Shorter text still obeys the selected output ceiling.

RawAssertion Output

Each extractor produces a list of RawAssertion records - pre-resolution claims with mention strings. This excerpt shows the main claim fields; the type also carries quote, entity-type, and event-lineage hints:
Assertions with significance: "skip" are recorded in the evidence layer but do not produce rels or attributes. This allows the extractor to note observations without polluting the knowledge graph.

Artifact Commit Boundary

After extraction, Meerkat Recall does not write graph rows directly from raw assertions. Instead:
  1. extract_assertions stages knowledge_candidates
  2. validate_artifacts validates those candidates
  3. commit_artifacts performs canonical writes
That commit phase is what creates assertions, relationships, attributes, and events:
1

Resolve Subject Entity

Look up by subject_entity_id (if the LLM found a match) or by identity key (lowercased display name). Identity-key reuse is type-guarded: all matches are fetched and the incoming type (normalized first) must be compatible with the candidate’s — topic is the unknown bucket and acts as a wildcard, upgrading to the specific incoming type on reuse. Ambiguous or type-incompatible matches route to an LLM disambiguator using the current space’s approved promotion_disambiguation binding; without one, incompatible types create a new entity rather than risking a false merge. Create if nothing matches.
2

Resolve Object Entity

Same process for entity-to-entity assertions. Skipped for literal-valued assertions.
3

Create Assertion Record

Always created in the evidence layer with the full evidence span and confidence score.
4

Create Rel or Attribute

If significance is not “skip”: creates a rel (entity-to-entity) or attribute (entity-to-literal) record.
5

Write Outbox Event

A batched outbox event is written with the created/updated record references. Truth maintenance picks this up asynchronously.

Diagnose an extraction failure

Use get_doc_pipeline_status on /mcp/full with the returned document ID, then inspect failed work items in the same space. Separate these cases before retrying: See ingesting data for a source-to-graph procedure and work queue recovery for retry semantics.