Models¶

llms¶

The llms section configures the individual model deployments. Each model has a unique key (e.g. azure-sweden-gpt-5) and the following properties:

deployment_name: The name of the deployment in the provider's system (e.g. gpt-5, eu.anthropic.claude-sonnet-4-6).
source: The model provider. See supported providers.
type (optional): The capability type of the deployment (text, image, audio, embedding, ...). See supported providers. Omitted for GoogleVertexAI.
cost_profile: The id of a profile in the cost_profiles section. Defines how requests to this model are priced.
api_key (optional): The API key for the provider. Can be omitted only for sources that use non-API-key auth, e.g. GoogleVertexAI (Workload Identity Federation).
priority (optional): Lets you prefer some models over others within a group. See load balancing.

Some fields apply only to certain sources:

url: Provider endpoint. Required for all OpenAI-based sources (AzureOpenAI, AzureAnthropic, GoogleVertexAI). Not used by AWS Bedrock sources.
api_version: Azure API version. Used by AzureOpenAI (and DeepSeek/embedding/image/audio deployments served through it).
region: AWS region. Required for all AwsBedrock* sources and must be in allowed_aws_regions.
region, sku (optional, Azure): the deployment's region (e.g. swedencentral) and sku (e.g. GlobalStandard). Used only to work out model residency; a model that leaves them unset has unknown residency.

Source reference¶

`source`	Required extra fields	Typical `type`
`AzureOpenAI`	`url`, `api_version`	`AzureOpenAIText`, `AzureOpenAIEmbedding`, `AzureOpenAIImage`, `AzureOpenAIAudio`, `OpenAITTSOld`, `AzureDeepSeekText`
`AzureAnthropic`	`url`	`GenericText`
`AwsBedrockAnthropic`	`region`	`AwsBedrockText`
`AwsBedrockNative`	`region`	`AwsBedrockText`
`AwsBedrockTitan`	`region`	`AwsBedrockText`
`GoogleVertexAI`	`url` (no `api_key`)	(omit)

Examples¶

One minimal example per source/type. The api_key value is resolved by prefix:

secret:<name> - looks up the secret by name in the configured secret store (preferred).
azure:<name> - same resolver, kept for backward compatibility with existing deployments.
plaintext:<value> - the raw key inline (use only for local runs/testing).

llms:
  # Azure OpenAI - text
  azure-sweden-gpt-5:
    deployment_name: gpt-5
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-sweden"
    cost_profile: azure-gpt-5
    api_version: "2025-04-01-preview"
    region: swedencentral     # for residency
    sku: DataZoneStandard     # for residency

  # Azure OpenAI - embeddings
  azure-sweden-text-embedding-3-large:
    deployment_name: text-embedding-3-large
    type: AzureOpenAIEmbedding
    source: AzureOpenAI
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-sweden"
    cost_profile: azure-text-embedding-3-large
    api_version: "2023-05-15"
    region: swedencentral
    sku: Standard

  # Azure OpenAI - image
  azure-sweden-gpt-image-1.5:
    deployment_name: gpt-image-1.5
    type: AzureOpenAIImage
    source: AzureOpenAI
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-sweden"
    cost_profile: azure-gpt-image-1.5
    api_version: "2025-04-01-preview"
    region: swedencentral
    sku: GlobalStandard

  # Azure OpenAI - audio (transcription)
  azure-eastus2-whisper:
    deployment_name: whisper
    type: AzureOpenAIAudio
    source: AzureOpenAI
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-eastus2"
    cost_profile: azure-whisper
    api_version: "2024-05-01-preview"
    region: eastus2
    sku: Standard

  # Azure OpenAI - legacy TTS (billed by input characters)
  azure-sweden-tts-hd:
    deployment_name: tts-hd
    type: OpenAITTSOld
    source: AzureOpenAI
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-sweden"
    cost_profile: azure-tts-hd
    api_version: "2025-03-01-preview"
    region: swedencentral
    sku: Standard

  # Azure DeepSeek (served through an Azure AI Foundry endpoint)
  azure-eastus2-deepseek-r1:
    deployment_name: DeepSeek-R1
    type: AzureDeepSeekText
    source: AzureOpenAI
    url: "https://<resource>.services.ai.azure.com"
    api_key: "secret:bss-llm-gateway-model-deepseek-r1-eastus2"
    cost_profile: azure-deepseek-r1
    api_version: "2024-05-01-preview"
    region: eastus2
    sku: GlobalStandard

  # Azure Anthropic (Claude on Azure, native Anthropic Messages API)
  azure-sweden-claude-sonnet-4-6:
    deployment_name: claude-sonnet-4-6
    type: GenericText
    source: AzureAnthropic
    url: "https://<resource>.openai.azure.com"
    api_key: "secret:bss-llm-gateway-model-foundry-swedencentral"
    cost_profile: "anthropic.claude-sonnet-4-6"
    region: swedencentral
    sku: DataZoneStandard

  # AWS Bedrock - Anthropic (Claude via the OpenAI-compatible path)
  "aws-eu-central-1-anthropic.claude-sonnet-4-6":
    deployment_name: "eu.anthropic.claude-sonnet-4-6"
    type: AwsBedrockText
    source: AwsBedrockAnthropic
    api_key: "secret:bss-llm-gateway-model-aws-bedrock"
    cost_profile: "anthropic.claude-sonnet-4-6"
    region: eu-central-1

  # AWS Bedrock - Native (Claude via the native Anthropic Messages API)
  "aws-eu-central-1-anthropic.claude-sonnet-4-6-native":
    deployment_name: "eu.anthropic.claude-sonnet-4-6"
    type: AwsBedrockText
    source: AwsBedrockNative
    api_key: "secret:bss-llm-gateway-model-aws-bedrock"
    cost_profile: "anthropic.claude-sonnet-4-6"
    region: eu-central-1

  # AWS Bedrock - Titan (Amazon native models)
  aws-eu-west-1-amazon.titan-text-express-v1:
    deployment_name: amazon.titan-text-express-v1
    type: AwsBedrockText
    source: AwsBedrockTitan
    api_key: "secret:bss-llm-gateway-model-aws-bedrock"
    cost_profile: amazon.titan-text-express-v1
    region: eu-west-1

  # Google Vertex AI (no api_key - uses Workload Identity Federation)
  google-eu-gemini-2.5-flash:
    deployment_name: google/gemini-2.5-flash
    source: GoogleVertexAI
    url: "https://europe-west1-aiplatform.googleapis.com/v1/projects/<project>/locations/europe-west1/endpoints/openapi"
    cost_profile: google-gemini-2.5-flash

llm_groups¶

The llm_groups section groups models that end users can select by a single name. Each group has a unique key (e.g. gpt-5) and a models list of model keys (from llms) that belong to the group.

Users call a group by its key; the gateway picks a model from the group (see load balancing). A group can hold deployments of the same model across several regions for failover:

llm_groups:
  gpt-5:
    models:
      - azure-sweden-gpt-5
  anthropic.claude-sonnet-4-6:
    models:
      - "aws-eu-central-1-anthropic.claude-sonnet-4-6"
      - "aws-eu-south-2-anthropic.claude-sonnet-4-6"

Example of selecting a group from the client:

client = create_azure_openai_client()
completion = client.chat.completions.create(
    model="gpt-5",  # the group id specified in the config 
    messages=[
        {
            "role": "user",
            "content": "Who is General Hammond from Stargate?",
        },
    ],
)
print(completion.choices[0].message.content)

cost_profiles¶

A cost profile says how to price usage for the models that reference it (through their cost_profile field). Each profile has:

id: The unique id used in a model's cost_profile (e.g. anthropic.claude-sonnet-4-6).
rate_lines: A list of pricing rules. Each rule prices one kind of usage.

Rate line fields¶

Each rate line has:

usage_dimension: What is being counted.
unit: The billing unit the price is expressed per.
price_usd: The price in USD per one unit. Written as a string to keep decimal precision.
extras (optional): Extra conditions that select this line for a special case (see extras options below). The line with no extras is the default price for that dimension.

usage_dimension values:

Value	Meaning
`input_tokens`	Standard input tokens
`output_tokens`	Generated output tokens
`reasoning_tokens`	Tokens spent on reasoning (thinking models)
`cache_read_tokens`	Tokens read from the prompt cache
`cache_write_tokens`	Tokens written to the prompt cache
`search_units`	Search units (e.g. a reranker)
`pages_processed`	Pages processed (e.g. document AI)

unit values:

Value	Price is per	Typical use
`tokens_1m`	1M tokens	Most text models
`tokens_1k`	1k tokens	Legacy token pricing
`minutes`	1 minute	Audio duration
`characters_1m`	1M characters	Legacy TTS
`search_unit_1k`	1k search units	Rerankers
`pages_processed_1k`	1k pages	Document AI

extras options:

token_tier: "basic" or "premium". Picks the long-context price. See Token tiers.
cache_creation_ttl: "5m" or "1h". Picks the price for a prompt-cache write with that TTL.
audio, image, text, duration: true flags for modality-specific lines (e.g. image output vs text output, or audio duration vs tokens).

Token tiers (premium long-context pricing)¶

Some providers charge more once a request's input grows past a size threshold. To model this, add a second rate line with extras: { token_tier: "premium" } for the higher price. The line without token_tier is the normal (basic) price.

The gateway tags a request as premium when its total input (input + cache read + cache write tokens) goes over:

200,000 tokens for Anthropic models (AzureAnthropic, AwsBedrockAnthropic, AwsBedrockNative).
272,000 tokens for OpenAI / Vertex models (OpenAI, AzureOpenAI, GoogleVertexAI) - only if the profile defines a premium line.

Below the threshold the basic price is used.

Examples¶

Simple per-token text model:

- id: azure-gpt-5
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "1.38" }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "11" }
    - { usage_dimension: "cache_read_tokens", unit: "tokens_1m", price_usd: "0.14" }

Embeddings (input only):

- id: azure-text-embedding-3-large
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "0.158" }

Token tiers + cache write TTLs (Anthropic Claude). The premium lines apply above 200k input tokens; the lines without token_tier are the default (basic) price:

- id: "anthropic.claude-sonnet-4-6"
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "3" }
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "6", extras: { "token_tier": "premium" } }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "15" }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "22.5", extras: { "token_tier": "premium" } }
    - { usage_dimension: "cache_read_tokens", unit: "tokens_1m", price_usd: "0.30" }
    - { usage_dimension: "cache_read_tokens", unit: "tokens_1m", price_usd: "0.60", extras: { "token_tier": "premium" } }
    - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "3.75", extras: { "cache_creation_ttl": "5m" } }
    - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "6", extras: { "cache_creation_ttl": "1h" } }

Image model. The image: true lines price image tokens; the lines without it price text tokens:

- id: azure-gpt-image-1.5
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "5.50" }
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "8.80", extras: { "image": "true" } }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "11" }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "35.20", extras: { "image": "true" } }

Google image model (gemini-2.5-flash-image). It reports a single output token count with no text/image split, so use a plain output_tokens line - not an image: "true" line, which would never match and bill $0:

- id: google-gemini-2.5-flash-image
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "0.30" }
    - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "30" }

Audio transcription (billed by audio duration):

- id: azure-whisper
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "minutes", price_usd: "0.0073", extras: { "audio": true, "duration": true } }

Legacy TTS (billed by input characters):

- id: azure-tts-hd
  rate_lines:
    - { usage_dimension: "input_tokens", unit: "characters_1m", price_usd: "36.30", extras: { "text": true } }

Image models¶

Image models report usage split into text tokens and image tokens. The gateway tags the image part with extras: { image: "true" }, so an image profile needs two output lines: a plain output_tokens line for the text part and an image: "true" line for the image part (same for input if the provider reports image input tokens). See the azure-gpt-image-1.5 profile in Examples. If the plain output line is missing, the text output bills $0.

These image models can be called and billed:

Model	`source` / `type`	Streaming	Non-streaming
`gpt-image-1.5`	`AzureOpenAI` / `AzureOpenAIImage`	Yes	Yes
`gpt-image-1`	`AzureOpenAI` / `AzureOpenAIImage`	Text output not billed	Text output not billed
`dall-e-3`	`AzureOpenAI` / `AzureOpenAIImage`	n/a	Yes (placeholder per-token price)
`gemini-2.5-flash-image`	`GoogleVertexAI`	Yes	Yes

Azure image models are called on /openai/v1/images/generations; gemini-2.5-flash-image on /openai/v1/chat/completions. Streaming and non-streaming are priced the same.

gpt-image-1 has no plain output line (no confirmed text-token price), so its text output bills $0 - use gpt-image-1.5 instead. gemini-2.5-flash-image returns one output count with no text/image split, so its profile uses a single plain output_tokens line priced at the image rate.

Legacy Format (Deprecated)¶

Note: The old format is still supported for backward compatibility but is deprecated and may be removed in a future version. We recommend migrating to the new rate_lines format.

The legacy format uses a flat structure:

id: The unique identifier of the cost profile.
usd_per_1k_input_tokens: Cost of 1000 input tokens in USD.
usd_per_1k_output_tokens: Cost of 1000 output tokens in USD.
usd_per_1k_cached_input_tokens (optional): Cost of 1000 cached input tokens in USD. Falls back to usd_per_1k_input_tokens if not set. Tracked for monitoring purposes only at the moment.

cost_profiles:
  - id: azure-gpt-4o
    usd_per_1k_input_tokens: 0.002
    usd_per_1k_output_tokens: 0.008
    usd_per_1k_cached_input_tokens: 0.001  # 50% of input price
  - id: free
    usd_per_1k_input_tokens: 0.0
    usd_per_1k_output_tokens: 0.0