Skip to content

Models

llms

The llms section contains the configuration for the model. Each model is defined by the unique key (e.g. azure-uk-gpt35), and the following properties:

  • deployment_name: The name of the deployment in the model provider's system.
  • source: The type of the model provider. List of supported providers can be found here.
  • type: Type of the source provider (text, image, video, ...). List of supported providers and types can be found here.
  • url: The URL of the model provider
  • api_key: The API key for the model provider.
  • cost_profile: The cost profile for the model. The cost profile defines the cost of the model. The cost profile is defined in the cost_profiles section.
  • priority (optional): Priorities allow you to prefer certain models over others in the selected group. Check load balancing section for more details.

llm_groups

The llm_groups section contains the configuration for the model groups. Each group is defined by the unique key (e.g. gpt35), and the following properties:

  • models: The list of models that belong to the group. Models must be defined in the llms section.

Groups are used by end users to select the model they want to use. The group is then used to select the model from the list of models. For example following code will select the model from the gpt35 group:

client = create_azure_openai_client()
completion = client.chat.completions.create(
    model="gpt35",
    messages=[
      {
          "role": "user",
          "content": "Who is General Hammond from Stargate?",
      },
    ],
)
print(completion.choices[0].message.content)

cost_profiles

The cost_profiles section contains the configuration for the cost profiles. Each cost profile is defined by the following properties:

  • id: The unique identifier of the cost profile (e.g., anthropic.claude-sonnet-4-20250514-v1:0).
  • rate_lines: A list of pricing rules, each containing:
    • usage_dimension: The type of token usage being priced. Possible values:
      • input_tokens: Standard input tokens
      • output_tokens: Generated output tokens
      • cache_read_tokens: Tokens read from cache
      • cache_write_tokens: Tokens written to cache
    • unit: The unit of measurement for pricing (e.g., tokens_1m for per 1 million tokens).
    • price_usd: The price in USD for the specified unit.
    • extras: Optional object containing additional conditions for the rate line:
      • token_tier: Token tier classification (e.g., "premium")
      • cache_creation_ttl: Cache time-to-live duration (e.g., "5m", "1h")

Notes

  • Multiple rate lines can exist for the same usage_dimension with different extras conditions to support tiered or conditional pricing.
  • When extras is omitted, the rate applies as the default for that usage dimension.
  • Prices are specified as strings to preserve decimal precision.

Legacy Format (Deprecated)

Note: The old format is still supported for backward compatibility but is deprecated and may be removed in a future version. We recommend migrating to the new rate_lines format.

The legacy format uses a flat structure with the following properties:

  • id: The unique identifier of the cost profile.
  • usd_per_1k_input_tokens: The cost of 1000 input tokens in USD.
  • usd_per_1k_output_tokens: The cost of 1000 output tokens in USD.
  • usd_per_1k_cached_input_tokens: Optional, the cost of 1000 cached output tokens in USD (tracked only for monitoring purposes at the moment)

Example of models configuration file:

llms:
  azure-uk-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-uk"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"

  azure-sweden-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-sweden"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"

  azure-us-text-embedding-3-large:
    deployment_name: text-embedding-3-large
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://models-cognitive-account-eastus2.openai.azure.com"
    api_key: "env:AZURE_OPENAI_API_KEY"
    cost_profile: azure-text-embedding-3-large

llm_groups:
  gpt-4.1:
    models:
      - azure-uk-gpt-4.1
      - azure-sweden-gpt-4.1
  text-embedding-3-large:
    models:
      - azure-us-text-embedding-3-large

cost_profiles:
  - id: azure-gpt-4.1
    usd_per_1k_input_tokens: 0.0022
    usd_per_1k_output_tokens: 0.0088
    usd_per_1k_cached_input_tokens: 0.0011  # 50% of input price
  - id: azure-text-embedding-3-large
    usd_per_1k_input_tokens: 0.00013
    usd_per_1k_output_tokens: 0.00013
  - id: anthropic.claude-sonnet-4-20250514-v1:0
    rate_lines:
      - { usage_dimension: "input_tokens", unit: "tokens_1m", price_usd: "3" }
      - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "15" }
      - { usage_dimension: "output_tokens", unit: "tokens_1m", price_usd: "22.5", extras: { "token_tier": "premium" } }
      - { usage_dimension: "cache_read_tokens", unit: "tokens_1m", price_usd: "0.30" }
      - { usage_dimension: "cache_read_tokens", unit: "tokens_1m", price_usd: "0.60", extras: { "token_tier": "premium" } }
      - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "3.75", extras: { "cache_creation_ttl": "5m" } }
      - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "7.5", extras: { "cache_creation_ttl": "5m", "token_tier": "premium" } }
      - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "6", extras: { "cache_creation_ttl": "1h" } }
      - { usage_dimension: "cache_write_tokens", unit: "tokens_1m", price_usd: "12", extras: { "cache_creation_ttl": "1h", "token_tier": "premium" } }