Skip to content

Configuration File

Configuration is a key part of the Adastra LLM Gateway (LLMGW). It allows you to define the models, groups, and other settings that are used by the LLMGW. There are two places for configuration:

  1. Using the yaml configuration file defined in the ./config folder
  2. Using the env variables, check the Environment Variables section to learn more.

The application configuration is defined in YAML files and is loaded by the LLMGW when it starts.

The configuration is divided into several sections, each of which defines a different part of the LLMGW's behavior. Configuration can be defined in multiple files, and the LLMGW will merge them together when it starts. Avoid defining the same configuration in multiple files, as this can lead to unexpected behavior.

Example of configuration file:

llms:
  azure-uk-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-uk"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"
    region: uksouth
    sku: GlobalStandard

  azure-sweden-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-sweden"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"
    region: swedencentral
    sku: DataZoneStandard

  azure-us-text-embedding-3-large:
    deployment_name: text-embedding-3-large
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://models-cognitive-account-eastus2.openai.azure.com"
    api_key: "env:AZURE_OPENAI_API_KEY"
    cost_profile: azure-text-embedding-3-large
    region: eastus2
    sku: Standard

  azure-us-cohere-rerank-v4.0-fast:
    deployment_name: cohere-rerank-v4.0-fast
    type: AzureOpenAIText
    source: AzureCohere
    url: "https://models-cognitive-account-eastus2.openai.azure.com"
    api_key: "env:AZURE_OPENAI_API_KEY"
    cost_profile: azure-us-cohere-rerank-v4.0-fast
    api_version: "2025-04-01-preview"
    region: eastus2
    sku: Standard

llm_groups:
  gpt-4.1:
    models:
      - azure-uk-gpt-4.1
      - azure-sweden-gpt-4.1
  text-embedding-3-large:
    models:
      - azure-us-text-embedding-3-large
  cohere-rerank-v4.0-fast:
    models:
      - azure-us-cohere-rerank-v4.0-fast

cost_profiles:
  - id: azure-gpt-4.1
    usd_per_1k_input_tokens: 0.0022
    usd_per_1k_output_tokens: 0.0088
    usd_per_1k_cached_input_tokens: 0.0011  # 50% of input price
  - id: azure-text-embedding-3-large
    usd_per_1k_input_tokens: 0.00013
    usd_per_1k_output_tokens: 0.00013
  - id: azure-us-cohere-rerank-v4.0-fast
    rate_lines:
      - { usage_dimension: "search_units", unit: "search_unit_1k", price_usd: "2.2" }

default_entity_types:
  - entity_type: project
    required: true
    allow_multiple: false
    spend_limit:
      amount: 100
      reset_period: DAY
  - entity_type: user
    required: true
    allow_multiple: false
    spend_limit:
      amount: 100.0
      reset_period: DAY
  - entity_type: groups
    required: false
    allow_multiple: true


retry_policy:
  max_attempts: 5
  time_limit_seconds: 600

circuit_breaker:
  failure_threshold: 5
  success_threshold: 5
  half_open_delay_in_secs: 30
  failure_counter_reset_in_secs: 30

health_check:
  enabled: True
  interval_seconds: 900
  health_check_prompt: "Translate 'Hello, how are you?' to French."
  excluded_groups: [
    "text-embedding-3-large",
  ]
  user: health-check
  project: health-check-project

residency:
  preferred: eu

Check the following sections to learn more about the configuration options: