Realtime Voice

Endpoint#

The console exposes the runtime's realtime WebSocket at wss://api.stimulir.com/api/v1/inference/realtime. Authenticate by sending Authorization: Bearer hyb_… on the upgrade request, the same hyb_* API key you use for chat completions. In BYOC deployments the runtime serves the raw endpoint directly:

Provider modes#

Realtime exposes a neutral voice session over WebSocket and bridges it to the active provider. Pick a provider with the provider query parameter; omit it to use the in-process HybrIE voice engine.

Managed (Gemini)

Append ?provider=gemini to route the session to Google's native-audio Gemini models. A typical model id is gemini-2.5-flash-native-audio-preview-12-2025. Billing falls under Managed Inference; voice minutes are metered separately from text tokens.

wss://api.stimulir.com/api/v1/inference/realtime?provider=gemini
# Upgrade header: Authorization: Bearer hyb_...
# Session model:  gemini-2.5-flash-native-audio-preview-12-2025

HybrIE-native (Qwen2.5-Omni)Experimental

Omit the provider query string and the session is served in-process by the HybrIE voice engine on the GPU node, currently Qwen2.5-Omni-3B. Configurable on the runtime with HYBRIE_REALTIME_MODEL. Qwen2.5-Omni is the same voice model used for the local /v1/audio/speech TTS endpoint.

wss://api.stimulir.com/api/v1/inference/realtime
# Upgrade header: Authorization: Bearer hyb_...
# Session model:  Qwen2.5-Omni-3B

Event shape#

Clients exchange JSON events over the socket. The schema is provider-neutral and bridged to the active provider's voice protocol, open a session, stream input audio frames, and receive output audio deltas. Shape only:

// client → server
{"type": "session.update", "session": {"model": "Qwen2.5-Omni-3B"}}
{"type": "input_audio_buffer.append", "audio": "<base64 pcm16>"}
{"type": "input_audio_buffer.commit"}

// server → client
{"type": "session.created"}
{"type": "response.output_audio.delta", "audio": "<base64 pcm16>"}
{"type": "response.done"}

Manual turn mode#

By default the active provider's server-side voice activity detection decides when a turn ends. Clients that run their own VAD can disable it and control turn boundaries explicitly. Append ?turn_detection=none to the WebSocket URL, or send a session.update with turn_detection set to {"type": "none"}.

wss://api.stimulir.com/api/v1/inference/realtime?provider=gemini&turn_detection=none

# or, per session:
{"type": "session.update", "session": {"turn_detection": {"type": "none"}}}

In manual mode the gateway brackets your audio appends with activity markers. The input_audio_buffer.commit ends the turn and triggers the response. Sessions that do not opt in keep the default server-side detection unchanged.

Python SDK#

The Stimulir SDK ships a first-class realtime client behind the realtime extra. It handles auth, project attribution (X-Project-Id from STIMULIR_PROJECT_ID), trace tags, turn detection, and the event protocol, you send PCM16 audio and iterate events:

pip install "stimulir[realtime]"

import asyncio
from stimulir.realtime import AudioDelta, RealtimeClient, ResponseDone, TextDelta

client = RealtimeClient(
    realtime_url="wss://api.stimulir.com/api/v1/inference/realtime?provider=gemini",
    api_key="hyb_...",              # or STIMULIR_API_KEY
    project_id="<project-id>",      # or STIMULIR_PROJECT_ID
    instructions="You are a concise voice assistant.",
    voice="Puck",                    # provider voice, optional
)

async def main() -> None:
    async with client.connect() as conn:
        await conn.setup()
        await conn.send_audio(pcm16_chunk)   # 16k mono PCM16 frames
        await conn.commit()                  # end the turn (manual mode)
        async for event in conn.events():
            if isinstance(event, AudioDelta):
                play(event.pcm16)            # 24k mono PCM16 out
            elif isinstance(event, TextDelta):
                print(event.text, end="")
            elif isinstance(event, ResponseDone):
                break

asyncio.run(main())

For client-side VAD, construct the client with turn_detection="none" (the SDK appends the URL parameter and brackets your audio with activity markers), or pass a {"type": "server_vad", ...} dict to tune the provider's server VAD. conn.send_context(text) injects text mid-session; conn.cancel() interrupts a response. See the Python SDK reference.

Metering#

Voice minutes are metered per session and surface as usage events with modality: "voice_realtime". Group billing by this field with /api/v1/usage/summary?group_by=modality, see Usage & Billing.