Live Channels

​

What this guide is for

• open a live audio/text channel on a session
• send audio, text, or model-supported image input to a live channel
• understand live channel lifecycle and capabilities
• reason about live channels in the context of the normal session model

​

Mental model

        +--------------------------+
        |   Session (history,      |
        |   tools, context)        |
        +--------------------------+
                     |
                     | session.model = "gpt-realtime-2"
                     v
        +--------------------------+
        | Live channel (caller-    |  audio/text at turn boundaries;
        | initiated via live/open) |  images after provider ACK +
        +--------------------------+  durable redacted receipt

• One canonical history. The session is the source of conversational truth. Committed audio and text join the same history as non-live turns at turn boundaries. A provider-acknowledged image is materialized and persisted as canonical context before the channel emits its redacted user_content_committed receipt.
• Capability gates channel open. ModelCapabilities.realtime is the signal that determines whether live/open succeeds. No channel is opened automatically.
• Channel lifecycle is caller-initiated. Call live/open to start a channel, live/close to end it, and live/send_input / live/commit_input / live/interrupt / live/truncate to control flow.
• Refresh without history replay. For ordinary OpenAI Realtime channels, live/refresh applies supported mutable settings such as visible tools. A canonical System-sequence change, model/provider swap, or canonical transcript or user-content-registry rewrite requires live/close + live/open. The adapter validates PCM at 24 kHz mono; refresh does not support arbitrary audio-format changes, and reopening does not make an unsupported format valid.

​

Realtime-capable models

• OpenAI Realtime: gpt-realtime-2 — the model for the ordinary shipped Realtime adapter, with still-image input advertised as capabilities.image_in == true.
• Public GPT Live: gpt-live-1 also advertises realtime == true, but requires the host-composed openai-live / execution-identity / WebRTC path described in GPT Live, not the Realtime WebSocket factory.
• Private experimental GPT Live: gpt-live-1-codex is a separate experimental catalog row on the deprecated experimental-gpt-live path, not an ordinary Realtime model.
• Gemini: reserved for future *-live* endpoints — no production models today.
• Anthropic: no realtime-capable models today.
• Self-hosted: realtime = false by default.

​

Opening a live channel

​

Create, then open

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "session/create",
  "params": {
    "prompt": "Let's talk.",
    "model": "gpt-realtime-2",
    "provider": "openai"
  }
}

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "live/open",
  "params": {
    "session_id": "01936f8a-7b2c-7000-8000-000000000001"
  }
}

from meerkat import MeerkatClient

client = MeerkatClient()
await client.connect(realm_id="team-alpha", live_ws=True)
try:
    session = await client.create_session(
        prompt="Let's talk.",
        model="gpt-realtime-2",
        provider="openai",
    )
    channel = await client.live_open(session.id)
finally:
    await client.close()

​

WebRTC SDP exchange

rkat-rpc --live-webrtc

rkat-rpc --live-ws 127.0.0.1:4900 --live-webrtc

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "live/open",
  "params": {
    "session_id": "01936f8a-7b2c-7000-8000-000000000001",
    "transport": "webrtc"
  }
}

{
  "channel_id": "ch_01936f8a-...",
  "transport": {
    "transport": "webrtc",
    "token": "single-use-bootstrap-token",
    "answer_method": "live/webrtc/answer"
  },
  "capabilities": {
    "audio_in": true,
    "audio_out": true,
    "text_in": true,
    "text_out": true,
    "image_in": true,
    "video_in": false,
    "transcript_supported": true,
    "barge_in_supported": true,
    "provider_native_resume": false
  },
  "continuity": {"mode": "fresh"}
}

async function waitForIceGatheringComplete(peer) {
  if (peer.iceGatheringState === "complete") return;
  await new Promise((resolve) => {
    function changed() {
      if (peer.iceGatheringState === "complete") {
        peer.removeEventListener("icegatheringstatechange", changed);
        resolve();
      }
    }
    peer.addEventListener("icegatheringstatechange", changed);
  });
}

const peer = new RTCPeerConnection();
const observations = peer.createDataChannel("meerkat.live");
const media = await navigator.mediaDevices.getUserMedia({ audio: true });
for (const track of media.getAudioTracks()) peer.addTrack(track, media);

const offer = await peer.createOffer();
await peer.setLocalDescription(offer);
await waitForIceGatheringComplete(peer);

const answered = await rpc("live/webrtc/answer", {
  channel_id: opened.channel_id,
  token: opened.transport.token,
  offer_sdp: peer.localDescription.sdp,
});
await peer.setRemoteDescription({ type: "answer", sdp: answered.answer_sdp });

​

Bounding the initial seed

{
  "jsonrpc": "2.0",
  "id": 2,
  "method": "live/open",
  "params": {
    "session_id": "01936f8a-7b2c-7000-8000-000000000001",
    "seed_max_chars": 24000
  }
}

result = await client.live_open(session.id, seed_max_chars=24_000)

const result = await client.liveOpen({
  session_id: session.id,
  seed_max_chars: 24_000,
});

const channel = LiveChannel.session(client, session.id, {
  seedMaxChars: 24_000,
});

​

Configuration defaults

[agent]
model = "gpt-realtime-2"

​

Observing channel status

{
  "jsonrpc": "2.0",
  "id": 3,
  "method": "live/status",
  "params": {"channel_id": "ch_01936f8a-..."}
}

​

Channel close events

​

Live channel methods

​

Live observations and durable history

​

Sending image input

{
  "jsonrpc": "2.0",
  "id": 4,
  "method": "live/send_input",
  "params": {
    "channel_id": "ch_01936f8a-...",
    "chunk": {
      "kind": "image",
      "idempotency_key": "turn-42-diagram",
      "mime": "image/png",
      "data": "iVBORw0KGgo..."
    }
  }
}

result = await client.live_open(session.id)
if not result["capabilities"]["image_in"]:
    raise RuntimeError("bound realtime model does not accept image input")

await client.live_send_input_image(
    result["channel_id"],
    "turn-42-diagram",
    "image/png",
    image_data_base64,
)
# `status: "sent"` is queue acceptance. Wait for the matching
# `user_content_committed` observation before sending dependent input.
# (Transport observation-loop code is omitted here.)
await client.live_send_input_text(
    result["channel_id"],
    "What is the dominant color in this image?",
)

const result = await client.liveOpen({ session_id: session.id });
if (!result.capabilities.image_in) {
  throw new Error("bound realtime model does not accept image input");
}

await client.liveSendInputImage(
  result.channel_id,
  "turn-42-diagram",
  "image/png",
  imageDataBase64,
);
// Resolution is queue acceptance only. Wait for the matching durable
// user_content_committed observation before sending dependent input.
// (Transport observation-loop code is omitted here.)

​

Acceptance, rejection, and durability

​

End-to-end example

from meerkat import MeerkatClient

client = MeerkatClient()
await client.connect(realm_id="team-alpha", live_ws=True)
try:

    # 1. Create a session on a realtime-capable model.
    session = await client.create_session(
        prompt="Ready for live audio.",
        model="gpt-realtime-2",
        provider="openai",
    )

    # 2. Open a live channel (returns channel_id + transport bootstrap).
    result = await client.live_open(
        session.id, transport="websocket", turning_mode="explicit_commit"
    )
    channel_id = result["channel_id"]
    # result["transport"] is the WebSocket URL + single-use token bootstrap
    # result["capabilities"] describes supported input/output modalities,
    # including model-gated image_in

    # 3. Send audio input and commit at turn boundaries.
    await client.live_send_input_audio(channel_id, audio_data, 24000, 1)
    await client.live_commit_input(channel_id)

    # 4. Close the channel when done.
    await client.live_close(channel_id)

    # 5. The session's history accumulates committed turns — same as a text session.
    history = await client.read_session_history(session.id)
finally:
    await client.close()

​

Live channels and mobs

id = "live-demo"

[profiles.host]
model = "gpt-realtime-2"
provider = "openai"
peer_description = "Live audio host"

[profiles.host.tools]
comms = true

​

GPT Live (public OpenAI Live API)

{"method":"live/open","params":{
  "session_id":"...",
  "transport":"webrtc",
  "execution_identity":{"version":"v1","profile_id":"openai.gpt-live-1.client-context.v1"}
}}

{"method":"mob/member_live_open","params":{
  "mob_id":"...",
  "agent_identity":"...",
  "transport":"webrtc",
  "execution_identity":{"version":"v1","profile_id":"openai.gpt-live-1.client-context.v1"}
}}

​

Generated context summaries for an existing agent

use meerkat::session_runtime::live_summary::LiveContextSummaryPolicy;
use std::time::Duration;

let policy = LiveContextSummaryPolicy::new(
    summary_service, // Arc<dyn LiveContextSummarizer>
    256 * 1024,      // maximum serialized input bytes
    8 * 1024,        // maximum returned UTF-8 bytes
    Duration::from_secs(20),
)?;
let member_host = member_host.with_context_summary_policy(policy);

​

Concurrent historical context

use meerkat::session_runtime::live_summary::LiveContextBootstrapMode;

let member_host = member_host.with_context_summary_policy(
    policy.with_bootstrap_mode(LiveContextBootstrapMode::Concurrent),
);

​

Delegate to the selected existing member

use meerkat_mob_mcp::live_delegation::{
    compose_experimental_live_delegation_coordinator_with_policy,
    LiveDelegationExecutionPolicy,
};

let coordinator = compose_experimental_live_delegation_coordinator_with_policy(
    runtime,
    mobs,
    LiveDelegationExecutionPolicy::ExistingMember,
);

​

Transcript rows while the member’s turn runs

​

Continuous media without fabricated playback completion

use meerkat::experimental_gpt_live::PublicGptLivePlaybackPolicy;

let authority = authority.with_public_playback_policy(
    PublicGptLivePlaybackPolicy::ProviderManagedUnmeasured,
)?;

​

Barge-in playback hint

• live/assistant_playback_hint { channel_id, hint: "duck" } is sent once when the user’s speech and audible assistant audio overlap, whichever side started: the user speaking over the assistant, or the assistant starting a response while the user keeps talking (at the user’s next voiced frame). A reply that starts after the user has stopped is never ducked.
• live/assistant_playback_hint { channel_id, hint: "restore" } follows when the assistant output goes quiet for 1.6 s (the response ended) or when the user goes quiet for 1.6 s (including a false trigger: the user stops without taking the turn), whichever comes first. On the user’s quiet, the restore depends on what the duck covers. A burst the user interrupted, one the assistant was voicing when the user started speaking, is restored at its next pause of 600 ms (at once if it is already pausing that long), so the stale tail the provider voices before it yields is not replayed. That burst ends at its first 600 ms pause, even while the user is still talking: audio after that pause, like any response that starts into or after the user’s speech, is an answer and is restored at once, also in the middle of a sentence, so the answer is heard. An answer spoken entirely while the user is still talking stays ducked to its end, because the user held the floor for all of it. Meerkat tells the two apart by onset order on the provider audio clock (gpt-live-1 output frames carry no response id). A restore follows every duck, so a duck never leaves playback muted for the next turn.

import { applyLiveAssistantPlaybackHint } from "@rkat/sdk";

// Server process: forward each hint to the page playing that channel.
client.onLiveNotification((notification) => {
  if (notification.method === "live/assistant_playback_hint") {
    sendToPage(notification.params); // { channel_id, hint }
  }
});

// Page: one gain node between the remote track and the speakers.
const gate = audioContext.createGain();
audioContext.createMediaStreamSource(remoteStream).connect(gate);
gate.connect(audioContext.destination);
function onPlaybackHint(params: { channel_id: string; hint: "duck" | "restore" }) {
  applyLiveAssistantPlaybackHint(gate, params.hint, audioContext.currentTime);
}

​

Media health of the first output

1. When that output’s first segment commits with a non-empty transcript, the runtime sends live/media_health_requested { channel_id, output_id }. The output_id is only a report key (that output is already settled; it is not a playback handle).
2. The client answers with live/media_health { channel_id, output_id, decoded_frames, audible_frames, max_rms }: raw counters from the channel’s media start to now (nothing was audible before the first output). audible_frames counts decoded frames whose RMS reached the client’s non-silent floor, and max_rms is linear amplitude (0.0 to 1.0). The client never sends a verdict.
3. The runtime judges the counters. audible keeps the channel open. A media_fault (no audible frame and a peak below 0.002) means the runtime has already closed the channel through the ordinary close (retained summary custody included) before it answered. reopen_recommended is true for the session’s first media fault, so the client can reopen with the retained context. A second fault on the same session closes without the recommendation, so a broken media path never loops.

​

Limitations and known gaps

• Ordinary/default composition uses OpenAI Realtime. Its provider integration is OpenAI’s Realtime API (gpt-realtime-2), not Azure OpenAI (azure_openai) or other providers. Public gpt-live-1 uses the separate host-composed openai-live / execution-identity / WebRTC path above; the private experimental path is deprecated.
• One live channel per session. A session has at most one live channel at a time. For per-member live channels in mobs, open channels against individual member sessions.
• Deferred sessions are model-gated. live/open may materialize a deferred session whose resolved model is realtime-capable; a deferred non-realtime session is rejected before channel creation. When the open materializes a deferred session, or opens one whose create-time prompt is still staged, the prompt reaches the realtime model as the session’s first user row: the run-start hooks admit it once, and a later text turn does not deliver it again. A hook that refuses it keeps it staged for the first text turn; the open goes ahead without it and reports staged_prompt_refused in the result’s advisories.
• Live rows wait for a running member turn. Transcript rows commit in arrival order behind a running turn of the member, so the persisted transcript can lag the conversation by that turn’s length (on a WebSocket or WebRTC channel, by each member turn queued behind it while turns run back to back; that channel’s voice and events keep flowing meanwhile); rows handed to a channel close are held in memory until they commit and are lost if the process exits first, and a row released mid-close can commit ahead of an earlier held row (see Transcript rows while the member’s turn runs).
• Ordinary Realtime refresh is not instruction/history replay. Supported mutable settings such as visible tools can refresh; canonical System-sequence changes, model/provider swaps, and canonical transcript or user-content-registry rewrites require live/close + live/open. The adapter remains fixed to PCM 24 kHz mono, including after reopen.
• GPT Live may delegate a question asked over its own playback. Public gpt-live-1 sessions use client delegation for the whole session, and the provider’s delegation events carry no typed reason, so Meerkat cannot keep a single turn native. A question that starts while the assistant is still speaking, before the cue for a delegated result has been sent, can be routed to the executor even when it asks about the conversation itself. The user then hears the answer late, in the same voice, once the executor’s turn has finished and its result has been spoken after at least 1.6 s of output silence; occasionally the voice instead resumes its earlier readout and never answers, and the question has to be asked again.

​

See also

• Mobs guide — spawning members, profiles, realtime-capable per-member models
• Configuration guide — setting agent.model
• JSON-RPC API — session/create, live/* methods
• REST API — REST routes for session creation