Skill Resolution Cascade
When a document enters the extraction pipeline, the system resolves two pieces of guidance: a purpose (what the space cares about) and a skill (domain-specific extraction instructions). Each is resolved by checking sources in priority order — the first non-empty value wins.Purpose Resolution
Skill Resolution
All current source types default to
extraction/technical-docs. In the normal
source-backed path, that fallback is reached before
PIPELINE_DEFAULT_EXTRACTION_SKILL. To change the default for source-backed
ingest, use an explicit space skill or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON
(including its "*" fallback); setting only the last environment default does
not override the source-type choice.
The skill resolution also performs filesystem fuzzy matching: skill IDs are normalized and matched against SKILL.md files in the skills directory (PIPELINE_SKILLS_DIR, default ./skills).
The resolution trace is recorded in the extraction plan artifact and exposed via the
get_doc_pipeline_status MCP endpoint for debugging which skill was selected and why.purpose and skill on the SpaceConfig - this applies to all documents ingested into that space without requiring per-document metadata.
Built-in skills
Meerkat Recall ships with domain-neutral extraction skills in theskills/extraction/ directory. Each is a SKILL.md file containing markdown guidance that gets injected into the LLM prompt. App-specific domains, including email/chat semantics, should provide their own skill directory and select it through metadata, space config, or PIPELINE_SKILL_BY_SOURCE_TYPE_JSON.
extraction/fiction - novels, stories, narratives
extraction/fiction - novels, stories, narratives
Entity types: person, location, device, group, topicFocus: Character relationships that persist across the narrative - who knows whom, who rules what, family ties, group memberships. Skips momentary actions (Alice picked up a bottle) in favor of durable state (Alice is the protagonist).Key predicates:
knows, related_to, parent_of, lives_in, rules_over, possesses, member_of, leads, has_role, has_traitextraction/technical-docs - architecture docs, specs, runbooks
extraction/technical-docs - architecture docs, specs, runbooks
Entity types: component, platform, concept, data_model, groupFocus: Component relationships, technology choices, architectural patterns, data model relationships, and design decisions. Targets the architecture-diagram level, not implementation code.Key predicates:
depends_on, part_of, communicates_with, uses, implements, stores_in, chosen_over, replacesSKILL.md file under skills/your-domain/. The skill ID is the directory path relative to the skills root (e.g., your-domain or extraction/your-domain).
Extraction Context
The extractor receives anExtractionContext containing everything it needs:
Extractors
Agentic Extractor (Meerkat-based)
The model defaults and summary policy on this page describe the current source checkout. Published 0.9.1 binaries retain their original defaults and summary behavior; using the newer defaults and policy requires the newer source. The primary extractor, enabled via themeerkat-extraction feature flag. It uses Meerkat - a modular LLM agent harness - to orchestrate extraction as a structured conversation with tool access.
- Model: configurable via
PIPELINE_AGENTIC_MODEL(binary and Docker Compose default:gpt-6.1-sol). Supply credentials and approved per-space profiles for the selected provider; these defaults do not migrate existing profiles. The pinned Meerkat client uses Responses for GPT-6.1 tool calls. - Reasoning: Governed extraction and support verification clear caller reasoning parameters before dispatch.
EXTRACTION_REASONING_EFFORTtherefore does not select their effective effort, and the current model-profile schema has no reasoning-effort field. GPT-6.1 Sol uses its upstreammediumdefault in this path; it supportslow,medium,high,xhigh, andmax, but notnoneorminimal. Selecting an explicit effort for governed extraction or support verification requires a product change. - Structured output: Meerkat receives a strict assertion schema and a bounded schema-repair retry count. Meerkat Recall parses and validates the result before staging candidates. Invalid output or an incomplete run can still fail the work item; a schema is not a guarantee of factual correctness.
- Tool access: the normal agentic path provides
ReadTextRangeDispatcherfor text-span checks andElephantToolBridgefor querying existing entities and predicates. Large text spans use a direct structured extraction path without those tools. An empty result from the agentic path can also trigger one direct structured retry. - Budget enforcement: the extractor uses the lower token and duration limits from its configuration and extraction context (defaults: 500,000 tokens and 600 seconds). The worker’s lifecycle timeout is a separate bound; see work queue configuration.
- Progress: the extraction call returns a completed result to the worker. Inspect document pipeline status and work-run logs for stage progress; this is not a public stream of partially committed assertions.
Non-Active Fallbacks
Meerkat Recall no longer uses a rule-based semantic fallback in the active runtime. If the real extractor is unavailable, semantic ingest is hidden and extraction work fails loudly with an explicit runtime/work-item error. Any non-LLM extraction helpers that remain in the codebase are test-only or legacy compatibility scaffolding, not part of the supported product path.Summary budgets
The space’s approvedsummary operation binding selects an immutable model
profile. Configure that profile’s summary policy when appending a new profile:
max_output_tokens is the provider’s total output ceiling, including thinking;
max_summary_length bounds the retained summary text in Unicode characters.
Accepted ranges are 1-65,536 tokens and 1-262,144 characters. Profiles without a
summary policy use runtime defaults of 4,096 tokens and 12,000 characters, with
the model-specific thinking policy described below. The local token estimate
can reduce the retained character ceiling further. These limits do not
guarantee that a model will use the full allowance or produce a complete answer.
For claude-haiku-5-5, adaptive_low sends adaptive thinking with low effort.
The pinned SDK cannot express Haiku’s supported disabled thinking mode, so
this policy does not promise zero thinking or a separate thinking-token cap.
For tool-free claude-sonnet-5-5 summaries, between_tools_low suppresses
up-front thinking and sends low effort. These model-specific policies apply
only to the named model on an Anthropic connection. provider_default omits explicit thinking controls. A profile with
an explicit summary policy may be bound only to the summary operation; use
separate profile IDs for other operations.
Only nonempty text with an SDK-reported end-of-turn completion is accepted as a
model summary. Empty text, token-limit stops, other known typed incomplete
completions, and provider errors use a bounded excerpt of the original text. Fallback and output
truncation are logged and counted, without logging source text. Provider
approval failures remain errors. The pinned SDK normalizes unknown raw stop
reasons, including model_context_window_exceeded, to end-of-turn; those
responses can escape incomplete-output detection. Preserving these raw reasons
is a tracked SDK follow-up. Inspect these counters under
WorkRunMetrics.custom.summary_process_totals.totals on completed work-run
metrics:
These are cumulative totals for the current process, not counts attributed to
one work run. One response can increment multiple counters, such as an empty
token-limited response followed by truncation of the local excerpt. The read-only Rust function
elephant_pipeline::summarizer::summary_metrics_snapshot() exposes the same
totals. Successful transport alone does not establish a useful summary.
Append a new profile ID and update the space’s summary binding through the
normal authorized processing-configuration replacement. Existing profiles are
immutable. During a historical migration, preserve the sealed configuration
through final catch-up; apply new-work model choices afterward. Summary policy
does not change embeddings or extraction/support-verification reasoning.
The summary trigger remains separate from the output allowance: input at or
above the configured minimum length (default 1,000 Unicode characters) invokes
the summarizer. Shorter text still obeys the selected output ceiling.
RawAssertion Output
Each extractor produces a list ofRawAssertion records - pre-resolution claims with mention strings. This excerpt shows the main claim fields; the type also carries quote, entity-type, and event-lineage hints:
Artifact Commit Boundary
After extraction, Meerkat Recall does not write graph rows directly from raw assertions. Instead:extract_assertionsstagesknowledge_candidatesvalidate_artifactsvalidates those candidatescommit_artifactsperforms canonical writes
1
Resolve Subject Entity
Look up by
subject_entity_id (if the LLM found a match) or by identity key (lowercased display name). Identity-key reuse is type-guarded: all matches are fetched and the incoming type (normalized first) must be compatible with the candidate’s — topic is the unknown bucket and acts as a wildcard, upgrading to the specific incoming type on reuse. Ambiguous or type-incompatible matches route to an LLM disambiguator using the current space’s approved promotion_disambiguation binding; without one, incompatible types create a new entity rather than risking a false merge. Create if nothing matches.2
Resolve Object Entity
Same process for entity-to-entity assertions. Skipped for literal-valued assertions.
3
Create Assertion Record
Always created in the evidence layer with the full evidence span and confidence score.
4
Create Rel or Attribute
If significance is not “skip”: creates a
rel (entity-to-entity) or attribute (entity-to-literal) record.5
Write Outbox Event
A batched outbox event is written with the created/updated record references. Truth maintenance picks this up asynchronously.
Diagnose an extraction failure
Useget_doc_pipeline_status on /mcp/full with the returned document ID, then
inspect failed work items in the same space. Separate these cases before retrying:
See ingesting data for a source-to-graph procedure and
work queue recovery for retry semantics.
