Routing overhead is measured, not hand-waved

The time spent choosing a model — classification plus catalog lookup, excluding the upstream provider call entirely — is measured on every "auto" request, returned in the x-llmrouter-routing-ms response header, and exported at GET /metrics as llmrouter_routing_overhead_ms, with a target of under 20ms. Pinning an explicit model id skips this step entirely — zero routing overhead, by construction.

Use streaming to see real TTFT

Time-to-first-token is only measurable on a streaming request — see Streaming. If perceived latency in an interactive UI matters more to you than total completion time, stream and render tokens as they arrive rather than waiting for the full response; real p50/p95 TTFT per model is visible on /usage and each model's own page.

Pass a session id for multi-turn conversations

Provider-side prompt caching only pays off if consecutive calls land on the same upstream node. Passing a stable session_id (body field or x-session-id header) for a given conversation raises the odds of that — see Provider prompt caching. It's a no-cost hint: omit it and routing behaves exactly as it does today.

The response cache is also a latency win

A response cache hit skips the upstream call completely — it isn't just cheaper, it's as fast as your own network round-trip to us. Worth opting into for any workload with genuinely repeated prompts (deterministic, low- or zero-temperature calls are the best fit).

Sort deployments by measured latency

If a model has more than one healthy deployment (direct + backstop, or multiple direct providers), setting provider.sort: "latency" — or the :nitro slug suffix — picks whichever deployment we've measured the lowest average latency for, instead of the default price-weighted pick. See Provider selection.