Sending an image

Use the same image_url content-part shape as the OpenAI API — a public URL (fetched by the provider, cheaper — no local encoding) or a base64 data URL (for local files or private/inaccessible images):

resp = client.chat.completions.create(
    model="auto",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}},
        ],
    }],
)

For a local file, base64-encode it into a data: URL instead of a public one:

import base64

with open("photo.jpg", "rb") as f:
    b64 = base64.b64encode(f.read()).decode()

resp = client.chat.completions.create(
    model="auto",
    messages=[{
        "role": "user",
        "content": [
            {"type": "text", "text": "What's in this image?"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{b64}"}},
        ],
    }],
)

Supported formats: image/png, image/jpeg, image/webp, and image/gif. Send multiple images by adding more image_url content parts to the same message — there's no platform-side limit, though the underlying model/provider may cap how many it'll actually look at. Put the text part before the image part(s) in the content array, as shown above — some models weight content earlier in the array more heavily, so images-then-text can read worse than text-then-images for the same prompt.

Capability-aware auto-routing

Under model:"auto", a message that actually contains an image is detected automatically — no flag to set — and only models the catalog marks vision-capable are considered. Pin an explicit model id and this check is skipped by default; a text-only model sent an image simply returns whatever error that provider gives for an unsupported input, the same as calling it directly. See Provider selection for require_parameters, which enforces the same capability check on an explicit pin too.

Which models support it

Filter by "Vision (image input)" at /models, or check the "Vision" badge on any model's own page. This flag comes from the same catalog sync described in Models, not a hand-maintained list.

Scope: images only, today

Only image input (image_url content parts) is recognized right now. Audio input, PDF/file input, and video are not yet supported — sending them isn't specially handled, so don't build against them yet.

Want an image back too, not just text? See Image output in chat — add modalities: ["image", "text"] to this same endpoint — or POST /v1/images for the dedicated generate-and-edit endpoint.