Chunk shape

stream = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Count to 5"}],
    stream=True,
)
for chunk in stream:
    if chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="")

Each event is a chat.completion.chunk object with an incremental delta, the stream ends with a literal data: [DONE] line — identical to the OpenAI API's own shape, regardless of which provider actually served the request underneath.

Usage on a stream

Token accounting works identically whether or not you ask for it: we always request stream_options.include_usage upstream so billing and the x-llmrouter-cost-usd header are correct either way — if you didn't ask for it yourself, we strip it back out of the final chunk before it reaches you, so response shape stays exactly what you requested.

Time-to-first-token

TTFT is only measurable on a streaming request — it's captured per request and feeds the latency numbers on /usage and each model's page, so you can see real p50/p95 time-to-first-token per model, not a marketing number.

Mid-stream errors

Failover across candidates (see Model fallbacks) only happens before the first token — once bytes are already streaming to you, a live upstream failure can't be transparently swapped for another model or provider. Instead it surfaces as a normal SSE error event carrying the request's trace_id, so you can look up exactly what happened at /logs. Tokens already delivered before the drop are still billed (and still yours) — see Zero-completion insurance.

Stream cancellation

Closing the connection from your side — aborting a fetch with an AbortController, breaking out of a for chunk in stream loop, or closing the underlying response — tears down the upstream provider connection too, so the request doesn't keep running server-side after you've stopped listening. Tokens already delivered before you disconnected are still billed and still recorded in the trace at /logs, the same as a mid-stream failure above — see Zero-completion insurance. This only applies to streaming requests — a non-streaming call has already been dispatched as a single blocking upstream request, so closing your connection early doesn't stop it or change what you're billed.

cancel.py 
import requests
from threading import Event, Thread

def stream_with_cancellation(prompt: str, cancel_event: Event):
    with requests.Session() as session:
        response = session.post(
            "https://llmrouter.sh/v1/chat/completions",
            headers={"Authorization": f"Bearer {api_key}"},
            json={"model": "auto", "messages": [{"role": "user", "content": prompt}], "stream": True},
            stream=True,
        )
        try:
            for line in response.iter_lines():
                if cancel_event.is_set():
                    response.close()
                    return
                if line:
                    print(line.decode(), end="", flush=True)
        finally:
            response.close()

# start the stream on a background thread, then cancel it whenever you like
cancel_event = Event()
stream_thread = Thread(target=lambda: stream_with_cancellation("Write a story", cancel_event))
stream_thread.start()

# ...later, to cancel:
cancel_event.set()

Headers on a stream

The same x-llmrouter-* headers described in Models & routing are sent with the initial response, before any bytes stream — with one exception: response-cache hit/miss status (x-openrouter-cache-status) is only ever set on non-streaming responses, since whether the first chunk came from cache isn't known until after headers are already committed. See Response caching.