Self-hosting models

Self-hosted models are configured as first-class Meerkat model IDs. Once an alias such as gemma-4-31b is registered, users can pass it anywhere they would pass a hosted model:

rkat run -m gemma-4-31b "Summarize this repository"
rkat run --resume -m gemma-4-e4b "Keep going, but use the faster local model"
rkat models
rkat doctor

This guide covers the general self-hosting contract and the Gemma 4 worked example. There is no separate Gemma page: Gemma 4 is one family under the same self-hosted model system as Ollama, LM Studio, vLLM, or another private OpenAI-compatible endpoint.

Model

Every self-hosted setup has three pieces:

Config section	Owns
`self_hosted.servers.<server_id>`	Transport, base URL, and API style for one serving endpoint
`self_hosted.models.<alias>`	The Meerkat-facing model ID, display metadata, capabilities, and upstream `remote_model`
`realm.<realm>.{backend,auth,binding}`	The credential binding for `provider = "self_hosted"`

The alias is the name users type. The remote_model is the model name returned or expected by the serving stack.

Server entries carry connection facts only. The legacy bearer_token / bearer_token_env server fields are rejected at config parse; credentials are owned by the realm auth profile selected through the binding. A self-hosted server with no realm binding fails closed at run time.

Server Shape

For Ollama, LM Studio, vLLM, and most private gateways, use the OpenAI-compatible transport:

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.servers.local]
transport = "openai_compatible"
base_url = "http://127.0.0.1:11434"
api_style = "chat_completions"

self_hosted.default_model must name one configured alias when more than one self_hosted.models entry exists. With exactly one alias it may be omitted and that alias becomes the default. Meerkat rejects an ambiguous multi-model configuration instead of choosing by table order. api_style = "chat_completions" is the conservative default for Gemma 4 and other self-hosted tool-calling models. Use another API style only after the specific serving stack has been validated with Meerkat tools, structured output, and multimodal input.

Realm Binding

Self-hosted credentials are realm-owned. Declare a backend, auth profile, and binding for provider = "self_hosted" in the same realm config, and make it the realm default (or select it per run with --auth-binding):

config.toml

[realm.global]
default_binding = "local"

[realm.global.backend.local]
provider = "self_hosted"
backend_kind = "self_hosted"
server = "local"                           # the [self_hosted.servers.<id>] this key authenticates

[realm.global.auth.local_auth]
provider = "self_hosted"
auth_method = "none"                       # or "api_key" / "static_bearer"
source = { kind = "platform_default" }     # unused for auth_method = "none"

[realm.global.binding.local]
backend_profile = "local"
auth_profile = "local_auth"

For a server that requires a bearer token, use:

config.toml

[realm.global.auth.local_auth]
provider = "self_hosted"
auth_method = "static_bearer"
source = { kind = "env", env = "LOCAL_LLM_TOKEN" }

Which Binding Authenticates Which Server

self_hosted is a provider class, not an endpoint identity: a private vLLM box and a hosted OpenAI-compatible gateway are both self_hosted and hold different secrets. Selecting a model therefore constrains credential resolution to the server that serves it - [self_hosted.models.<alias>].server - and not to whichever realm on the chain happens to declare a self-hosted default. A backend identifies its endpoint in one of two ways:

Declaration	Meaning
`server = "<server_id>"` on the backend profile	Canonical. This binding authenticates that server.
`default_model` on the binding	Inferred: the binding authenticates the server that serves its default model. Used for configs written before `server` existed.

A binding that declares neither is unconstrained: it is used only when it is the single unconstrained self-hosted binding reachable from the active realm, so one-server setups keep working with no annotation. When the server cannot be identified honestly - no binding names it, or several unconstrained bindings are reachable - the run fails closed with an error naming the server and every binding considered, rather than sending some other server’s key and surfacing that endpoint’s Unauthorized. Fix it by adding server = "<server_id>" to the backend profile that authenticates it, or by selecting the binding explicitly with --auth-binding <realm>:<binding>. An explicit selection wins, except when the named binding declares a different server: that contradiction fails closed too.

Alias Shape

config.toml

[self_hosted.models.gemma-4-31b]
server = "local"
remote_model = "gemma4:31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
supports_web_search = false
call_timeout_secs = 600

vision = true advertises image input. It is separate from image tool-result blocks: the registry forces image_tool_results = false for api_style = "chat_completions"; only a validated responses endpoint may advertise that result shape. supports_thinking and supports_reasoning describe the behavior Meerkat should expose through the configured transport. Gemma 4 is reasoning-capable, but normalized reasoning controls and trace streaming still vary by serving stack, so validate the behavior you plan to rely on. Keep supports_web_search = false. The current self-hosted factory path has no provider-native web-search lowering. Setting it to true would suppress the fallback without creating a native executor, so search requests would have no working path. An explicit Meerkat web-search enable fails closed when no fallback executor can be provisioned.

Meerkat’s current self-hosted path does not expose self-hosted Gemma audio or a self-hosted realtime transport. Treat these aliases as text, image-input, and tool-capable models unless a dedicated self-hosted realtime path is documented.

Gemma 4 Aliases

Recommended aliases:

Alias	Good default use
`gemma-4-e2b`	Lowest-footprint local experiments
`gemma-4-e4b`	Fast local iteration with more headroom
`gemma-4-26b-a4b`	Stronger quality on a serious local or remote GPU setup
`gemma-4-31b`	Best quality of the four, usually best on a dedicated server

Ollama

Use Ollama when the model runs on the same machine as Meerkat and you want the lightest local setup.

Serve the model

ollama pull gemma4:31b
ollama list

config.toml

[self_hosted.servers.ollama]
transport = "openai_compatible"
base_url = "http://127.0.0.1:11434"
api_style = "chat_completions"

Add aliases

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.models.gemma-4-e2b]
server = "ollama"
remote_model = "gemma4:e2b"
display_name = "Gemma 4 E2B"
family = "gemma-4"
tier = "supported"
context_window = 128000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true

[self_hosted.models.gemma-4-31b]
server = "ollama"
remote_model = "gemma4:31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

LM Studio

Use LM Studio when you want a desktop-managed OpenAI-compatible server.

Start the local server

Load the Gemma 4 model in LM Studio, then start the local server.

config.toml

[self_hosted.servers.lmstudio]
transport = "openai_compatible"
base_url = "http://127.0.0.1:1234"
api_style = "chat_completions"

Alias the served model

Use the model name LM Studio exposes in its /v1/models output.

config.toml

[self_hosted.models.gemma-4-e4b]
server = "lmstudio"
remote_model = "google/gemma-4-e4b"
display_name = "Gemma 4 E4B"
family = "gemma-4"
tier = "supported"
context_window = 128000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true

vLLM

Use vLLM when you want a private server with more deployment control.

Launch vLLM

Start vLLM with the Gemma 4 model you want to expose.

config.toml

[self_hosted.servers.vllm]
transport = "openai_compatible"
base_url = "http://my-gpu-box:8000"
api_style = "chat_completions"

If the endpoint requires a token, point the realm auth profile at it:

config.toml

[realm.dev.auth.local_auth]
provider = "self_hosted"
auth_method = "static_bearer"
source = { kind = "env", env = "VLLM_API_TOKEN" }

Add remote aliases

Point aliases at the exact model names your vLLM endpoint exposes.

config.toml

[self_hosted]
default_model = "gemma-4-31b"

[self_hosted.models.gemma-4-26b-a4b]
server = "vllm"
remote_model = "google/gemma-4-26b-a4b"
display_name = "Gemma 4 26B A4B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

[self_hosted.models.gemma-4-31b]
server = "vllm"
remote_model = "google/gemma-4-31b"
display_name = "Gemma 4 31B"
family = "gemma-4"
tier = "supported"
context_window = 256000
max_output_tokens = 8192
vision = true
image_tool_results = false
inline_video = false
supports_temperature = true
supports_thinking = true
supports_reasoning = true
call_timeout_secs = 600

Validation

Run these after adding or editing self-hosted model config:

rkat models
rkat doctor
rkat auth bindings --realm global
rkat auth test --realm global local
rkat run -m gemma-4-31b "Say hello in one sentence."

The expected result:

rkat models shows a self_hosted provider group and the aliases you added.
rkat doctor resolves the realm binding and reports the server as reachable.
rkat auth bindings shows which backend and auth profile the local binding joins, and rkat auth test local resolves it without running a model turn.
rkat run -m ... works without an explicit --provider.
The alias points at the exact upstream remote_model exposed by the server.

If rkat run fails with “no credential binding for self-hosted server”, add the realm binding shown above, and give its backend profile the server = "<server_id>" line so it names the endpoint it authenticates.

Providers

Hosted and self-hosted provider model.

CLI configuration

Config file locations and model settings.

CLI commands

Commands for running, diagnosing, and inspecting models.

Getting started

Concepts

Guides: Models And Hosting

Guides: Runtime Behavior

Guides: Orchestration

Examples: Foundations

Examples: Capabilities

Examples: Orchestration And Deployment

Self-hosting models

Model

Server Shape

Realm Binding

Which Binding Authenticates Which Server

Alias Shape

Gemma 4 Aliases

Ollama

LM Studio

vLLM

Validation

See Also

Providers

CLI configuration

CLI commands

​Model

​Server Shape

​Realm Binding

​Which Binding Authenticates Which Server

​Alias Shape

​Gemma 4 Aliases

​Ollama

​LM Studio

​vLLM

​Validation

​See Also

Providers

CLI configuration

CLI commands

Model

Server Shape

Realm Binding

Which Binding Authenticates Which Server

Alias Shape

Gemma 4 Aliases

Ollama

LM Studio

vLLM

Validation

See Also