Self-hosting models

Self-hosted models are configured as first-class Meerkat model IDs. Once an alias such as gemma-4-31b is registered, users can pass it anywhere they would pass a hosted model:

rkat run -m gemma-4-31b "Summarize this repository"
rkat run --resume -m gemma-4-e4b "Keep going, but use the faster local model"
rkat models
rkat doctor

This guide covers the general self-hosting contract and the Gemma 4 worked example. There is no separate Gemma page: Gemma 4 is one family under the same self-hosted model system as Ollama, LM Studio, vLLM, or another private OpenAI-compatible endpoint. The Ollama, LM Studio, and vLLM recipes below are alternatives. Put the chosen server and alias tables plus the common realm binding in the default global config document, ~/.rkat/config.toml, so the workspace can inherit them. Replace the generic server/alias examples with the chosen recipe rather than combining every example, and run only commands whose aliases you registered.

Model

Every self-hosted setup has three pieces:

Config section	Owns
`self_hosted.servers.<server_id>`	Transport, base URL, and API style for one serving endpoint
`self_hosted.models.<alias>`	The Meerkat-facing model ID, display metadata, capabilities, and upstream `remote_model`
`realm.<realm>.{backend,auth,binding}`	The credential binding for `provider = "self_hosted"`

The alias is the name users type. The remote_model is the model name returned or expected by the serving stack.

Server entries carry connection facts only. The legacy bearer_token / bearer_token_env server fields are rejected at config parse; credentials are owned by the realm auth profile selected through the binding. A self-hosted server with no realm binding fails closed at run time.

Server Shape

For Ollama, LM Studio, vLLM, and most private gateways, use the OpenAI-compatible transport:

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.servers.local]
transport = "openai_compatible"
base_url = "http://127.0.0.1:11434"
api_style = "chat_completions"

self_hosted.default_model must name one configured alias when more than one self_hosted.models entry exists. With exactly one alias it may be omitted and that alias becomes the default. Meerkat rejects an ambiguous multi-model configuration instead of choosing by table order. api_style = "chat_completions" is the conservative default for Gemma 4 and other self-hosted tool-calling models. Use another API style only after the specific serving stack has been validated with Meerkat tools, structured output, and multimodal input.

Realm Binding

Self-hosted credentials are realm-owned. Declare a backend, auth profile, and binding for provider = "self_hosted" in the same realm config, and make it the realm default (or select it per run with --auth-binding):

config.toml

[realm.global]
default_binding = "local"

[realm.global.backend.local]
provider = "self_hosted"
backend_kind = "self_hosted"
server = "local"                           # the [self_hosted.servers.<id>] this key authenticates

[realm.global.auth.local_auth]
provider = "self_hosted"
auth_method = "none"                       # or "api_key" / "static_bearer"
source = { kind = "platform_default" }     # unused for auth_method = "none"

[realm.global.binding.local]
backend_profile = "local"
auth_profile = "local_auth"

For a server that requires a bearer token, use:

config.toml

[realm.global.auth.local_auth]
provider = "self_hosted"
auth_method = "static_bearer"
source = { kind = "env", env = "LOCAL_LLM_TOKEN" }

Which Binding Authenticates Which Server

self_hosted is a provider class, not an endpoint identity: a private vLLM box and a hosted OpenAI-compatible gateway are both self_hosted and hold different secrets. Selecting a model therefore constrains credential resolution to the server that serves it - [self_hosted.models.<alias>].server - and not to whichever realm on the chain happens to declare a self-hosted default. A backend identifies its endpoint in one of two ways:

Declaration	Meaning
`server = "<server_id>"` on the backend profile	Canonical. This binding authenticates that server.
`default_model` on the binding	Inferred: the binding authenticates the server that serves its default model. Used for configs written before `server` existed.

A binding that declares neither is unconstrained: it is used only when it is the single unconstrained self-hosted binding reachable from the active realm, so one-server setups keep working with no annotation. A binding whose auth profile runs a credential command (source = { kind = "command" }) must declare server: the command’s token is the host’s, and only the declaration says where it goes. A realm with such a binding that declares none does not compose. The binding is never inferred from its default_model and never unconstrained, and naming it explicitly for a model it does not declare fails closed. When the server cannot be identified honestly - no binding names it, or several unconstrained bindings are reachable - the run fails closed with an error naming the server and every binding considered, rather than sending some other server’s key and surfacing that endpoint’s Unauthorized. Fix it by adding server = "<server_id>" to the backend profile that authenticates it, or by selecting the binding explicitly with --auth-binding <realm>:<binding>. An explicit selection wins, except when the named binding declares a different server, or is a credential command binding that declares none: those fail closed too.

Alias Shape

config.toml

[self_hosted.models.gemma-4-31b]
server = "local"
remote_model = "gemma4:31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
supports_web_search = false
call_timeout_secs = 600
stream_inactivity_timeout_secs = 600

call_timeout_secs bounds one model call. stream_inactivity_timeout_secs sets how long this model’s stream may stay silent, for example during a long prefill before the first token, before Meerkat aborts the call as stalled and retries it. It must be at least 60 seconds: the client waits up to 30 seconds after the model’s final event for the usage trailer, and a shorter window would race that wait, so config validation refuses it. Omit it to use the [retry] stream_inactivity_timeout window (300 seconds by default); an explicit [retry] stream_inactivity_timeout applies to every model, this one included. vision = true advertises image input. It is separate from image tool-result blocks: the registry forces image_tool_results = false for api_style = "chat_completions"; only a validated responses endpoint may advertise that result shape. supports_thinking and supports_reasoning describe the behavior Meerkat should expose through the configured transport. Gemma 4 is reasoning-capable, but normalized reasoning controls and trace streaming still vary by serving stack, so validate the behavior you plan to rely on. Keep supports_web_search = false. The current self-hosted factory path has no provider-native web-search lowering. Setting it to true would suppress the fallback without creating a native executor, so search requests would have no working path. An explicit Meerkat web-search enable fails closed when no fallback executor can be provisioned.

Meerkat’s current self-hosted path does not expose self-hosted Gemma audio or a self-hosted realtime transport. Treat these aliases as text, image-input, and tool-capable models unless a dedicated self-hosted realtime path is documented.

Gemma 4 Aliases

Recommended aliases:

Alias	Good default use
`gemma-4-e2b`	Lowest-footprint local experiments
`gemma-4-e4b`	Fast local iteration with more headroom
`gemma-4-26b-a4b`	Stronger quality on a serious local or remote GPU setup
`gemma-4-31b`	Best quality of the four, usually best on a dedicated server

Ollama

Use Ollama when the model runs on the same machine as Meerkat and you want the lightest local setup.

Serve the model

ollama pull gemma4:31b
ollama list

In the common [realm.global.backend.local] binding backend, change server = "local" to server = "ollama". Keep the binding named local and its local_auth profile in global.

config.toml

[self_hosted.servers.ollama]
transport = "openai_compatible"
base_url = "http://127.0.0.1:11434"
api_style = "chat_completions"

Add aliases

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.models.gemma-4-e2b]
server = "ollama"
remote_model = "gemma4:e2b"
display_name = "Gemma 4 E2B"
family = "gemma-4"
tier = "supported"
context_window = 128000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true

[self_hosted.models.gemma-4-31b]
server = "ollama"
remote_model = "gemma4:31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

LM Studio

Use LM Studio when you want a desktop-managed OpenAI-compatible server.

Start the local server

Load the Gemma 4 model in LM Studio, then start the local server.

In the common [realm.global.backend.local] binding backend, change server = "local" to server = "lmstudio". Keep the binding named local and its local_auth profile in global.

config.toml

[self_hosted.servers.lmstudio]
transport = "openai_compatible"
base_url = "http://127.0.0.1:1234"
api_style = "chat_completions"

Alias the served model

Use the model name LM Studio exposes in its /v1/models output.

config.toml

[self_hosted]
default_model = "gemma-4-e4b"

[self_hosted.models.gemma-4-e4b]
server = "lmstudio"
remote_model = "google/gemma-4-e4b"
display_name = "Gemma 4 E4B"
family = "gemma-4"
tier = "supported"
context_window = 128000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true

vLLM

Use vLLM when you want a private server with more deployment control.

Launch vLLM

Start vLLM with the Gemma 4 model you want to expose.

In the common [realm.global.backend.local] binding backend, change server = "local" to server = "vllm". Keep the binding named local.

config.toml

[self_hosted.servers.vllm]
transport = "openai_compatible"
base_url = "http://my-gpu-box:8000"
api_style = "chat_completions"

If the endpoint requires a token, replace the common global auth profile with this one and set VLLM_API_TOKEN before running Meerkat:

config.toml

[realm.global.auth.local_auth]
provider = "self_hosted"
auth_method = "static_bearer"
source = { kind = "env", env = "VLLM_API_TOKEN" }

Add remote aliases

Point aliases at the exact model names your vLLM endpoint exposes.

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.models.gemma-4-26b-a4b]
server = "vllm"
remote_model = "google/gemma-4-26b-a4b"
display_name = "Gemma 4 26B A4B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

[self_hosted.models.gemma-4-31b]
server = "vllm"
remote_model = "google/gemma-4-31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

Refusals and Content Filters

Since 0.9.4 a refusal from an OpenAI-compatible server ends the run the way an OpenAI refusal does: a streamed delta.refusal ends the response as a refusal, and a finish_reason of content_filter ends it as a content filter. The run fails as content_filtered with that stop and an open_ai-tagged detail, and the refusal text is never committed as an answer. See Refusals and content filters.

Validation

Run these after adding or editing self-hosted model config:

rkat models
rkat doctor
rkat auth bindings --realm global
rkat auth test --realm global local

Then run the model that the selected recipe defines:

# Ollama or vLLM recipe:
rkat run --auth-binding global:local -m gemma-4-31b "Say hello in one sentence."

# LM Studio recipe:
rkat run --auth-binding global:local -m gemma-4-e4b "Say hello in one sentence."

Here local is the binding ID, not necessarily the server ID. Its backend’s server must match the selected recipe (ollama, lmstudio, or vllm) and the alias’s server. The expected result:

rkat models shows a self_hosted provider group and the aliases you added.
rkat doctor resolves the realm binding and reports the server as reachable.
rkat auth bindings shows which backend and auth profile the local binding joins, and rkat auth test local resolves it without running a model turn.
rkat run -m ... works without an explicit --provider.
The alias points at the exact upstream remote_model exposed by the server.

If rkat run fails with “no credential binding for self-hosted server”, add the realm binding shown above, and give its backend profile the server = "<server_id>" line so it names the endpoint it authenticates.

Providers

Hosted and self-hosted provider model.

CLI configuration

Config file locations and model settings.

CLI commands

Commands for running, diagnosing, and inspecting models.

Getting started

Concepts

Guides: Models And Hosting

Guides: Runtime Behavior

Guides: Orchestration

Examples: Foundations

Examples: Capabilities

Examples: Orchestration And Deployment

Self-hosting models

Model

Server Shape

Realm Binding

Which Binding Authenticates Which Server

Alias Shape

Gemma 4 Aliases

Ollama

LM Studio

vLLM

Refusals and Content Filters

Validation

See Also

Providers

CLI configuration

CLI commands

​Model

​Server Shape

​Realm Binding

​Which Binding Authenticates Which Server

​Alias Shape

​Gemma 4 Aliases

​Ollama

​LM Studio

​vLLM

​Refusals and Content Filters

​Validation

​See Also

Providers

CLI configuration

CLI commands

Model

Server Shape

Realm Binding

Which Binding Authenticates Which Server

Alias Shape

Gemma 4 Aliases

Ollama

LM Studio

vLLM

Refusals and Content Filters

Validation

See Also