Configuration file

Configuration is a key part of the Adastra LLM Gateway (LLMGW). It allows you to define the models, groups, and other settings that are used by the LLMGW. There are two places for configuration:

  1. Using the yaml configuration file defined in the ./config folder
  2. Using the env variables, check the Environment Variables section to learn more.

The application configuration is defined in YAML files and is loaded by the LLMGW when it starts.

The configuration is divided into several sections, each of which defines a different part of the LLMGW’s behavior. Configuration can be defined in multiple files, and the LLMGW will merge them together when it starts. It is not recommended to define the same configuration in multiple files, as this can lead to unexpected behavior.

Example of configuration file:

llms:
  azure-uk-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-uk"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"

  azure-sweden-gpt-4.1:
    deployment_name: gpt-4.1
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://bss-llm-gateway-model-models-cognitive-account-uk.openai.azure.com"
    api_key: "azure:https://bss-llm-gateway-test.vault.azure.net/secrets/bss-llm-gateway-model-sweden"
    cost_profile: azure-gpt-4.1
    api_version: "2025-01-01-preview"

  azure-us-text-embedding-3-large:
    deployment_name: text-embedding-3-large
    type: AzureOpenAIText
    source: AzureOpenAI
    url: "https://models-cognitive-account-eastus2.openai.azure.com"
    api_key: "env:AZURE_OPENAI_API_KEY"
    cost_profile: azure-text-embedding-3-large

llm_groups:
  gpt-4.1:
    models:
      - azure-uk-gpt-4.1
      - azure-sweden-gpt-4.1
  text-embedding-3-large:
    models:
      - azure-us-text-embedding-3-large

cost_profiles:
  - id: azure-gpt-4.1
    usd_per_1k_input_tokens: 0.0022
    usd_per_1k_output_tokens: 0.0088
    usd_per_1k_cached_input_tokens: 0.0011  # 50% of input price
  - id: azure-text-embedding-3-large
    usd_per_1k_input_tokens: 0.00013
    usd_per_1k_output_tokens: 0.00013

default_entity_types:
  - entity_type: project
    required: true
    allow_multiple: false
    spend_limit:
      amount: 100
      reset_period: DAY
  - entity_type: user
    required: true
    allow_multiple: false
    spend_limit:
      amount: 100.0
      reset_period: DAY
  - entity_type: groups
    required: false
    allow_multiple: true


retry_policy:
  max_attempts: 5
  time_limit_seconds: 600

circuit_breaker:
  failure_threshold: 5
  success_threshold: 5
  half_open_delay_in_secs: 30
  failure_counter_reset_in_secs: 30

health_check:
  enabled: True
  interval_seconds: 900
  health_check_prompt: "Translate 'Hello, how are you?' to French."
  excluded_groups: [
    "text-embedding-3-large",
  ]
  user: health-check
  project: health-check-project

Check the following sections to learn more about the configuration options: