Skip to content

LLMGW container

This page lists the environment variables supported by the LLMGW container. Variables are grouped by area and include defaults and usage notes where applicable.

App

  • DEBUG Whether the app is in debug mode. Mainly affects logging and telemetry.
    • Default is false.
  • ENVIRONMENT Deployment environment.
    • Supported values: dev, test, prd, local.
  • SESSION_SECRET_KEY The key used to protect FastAPI/Starlette session cookies. Should be at least 256 bits in production.
  • ADMIN_API_KEY Admin API key that can be used for access to the admin portal.
  • ALLOWED_HOSTS A comma-separated list of hosts against which the incoming request's Host header will be checked.
    • Example: ["localhost", "my-test-domain.net"]
  • ADMIN_PORTAL_BRAND_COLOR Brand color (hex color code for primary brand color).
  • ADMIN_PORTAL_BRAND_ICON Brand icon (base64-encoded PNG for favicon).
  • ADMIN_PORTAL_BRAND_LOGO Brand logo (base64-encoded PNG for sidebar header).

Configuration sources

  • CONFIG_SOURCES A JSON list of configuration source definitions. Always in a format of *prefix*:*service-specific source identifier*. Allowed prefices are dir, aws_s3 and azure_blob..
    All sources must use the same backend, e.g., s3: cannot be mixed with azureblob:. For Azure Blob Storage the format is <container>/<blob>; for AWS S3 <bucket>/<file_path>; for local directory a filepath.

Examples: 

[
  "dir:/etc/llmgw",
  "dir:./config"
]

[
  "s3:config-bucket/config-prod.yaml",
  "s3:config-bucket/extra-models.yaml"
]

Entities

  • ALLOW_ANONYMOUS_ENTITIES Allow requests without a defined entity (user/project).
    • Default: true (helpful when spend limits are not enforced).

Secrets and secret stores

  • SECRET_STORE Secret provider backend used to resolve the secrets.

    • Supported values: env, azure_kv, aws_secrets_manager.

  • APPLICATION_KEY_VAULT_URL Azure Key Vault URL used when SECRET_STORE=azure_kv.

    • Example: https://llmgw-kv.vault.azure.net/

  • ENV_SERVICE_CACHE_EXPIRATION_SECONDS Cache expiration time (seconds) for local secrets.

    • Default: 600

Storage backends

  • STORAGE Storage backend used to store/load the YAML config.

    • Supported values: dir, azure_blob, aws_s3.

  • AZURE_BLOB_STORAGE_ACCOUNT_URL Azure Blob Storage account URL (required when STORAGE=azure_blob).

Database

  • DB_CONNECTION_STRING The Postgres connection string in libpq (key=value) or URL style.

OTEL

  • OTEL_SDK_DISABLED Disable OpenTelemetry configuration.

    • Default: false (OTEL enabled)

  • OTEL_TRACES_EXPORTER OTEL Traces exporter.

    • Supported values: otlp, console, or none.
    • Default: otlp.

  • OTEL_METRICS_EXPORTER OTEL Metrics exporter.

    • Supported values: otlp, console, or none.
    • Default: otlp.

  • OTEL_LOGS_EXPORTER OTEL Logs exporter.

    • Supported values: otlp or none.
    • Default: otlp.

  • OTEL_EXPORTER_ENDPOINT OTLP exporter endpoint. Required if any of OTEL_*_EXPORTER is otlp.

    • Example: http://<SERVICE_HOST>:4318
    • Azure Container Apps: <SERVICE_HOST> is the ACA service name (e.g., bss-llm-gateway-dev-otel).

  • OTEL_PERIODIC_EXPORT_INTERVAL_MILLIS OTEL periodic export interval (ms).

    • Default: 60000

  • DEBUG_METRICS When main exporter is otlp, enable additional console exporters / debug messages.

    • Boolean flag: 1, true, True.

Admin portal and SSO

  • ADMIN_AUTH_ENABLED Enable admin authentication/SSO. Warning: disabling this protection and removes SSO auth endpoints.

  • ADMIN_SSO_CLIENT_ID OAuth client ID (e.g., Entra ID App ID).

  • ADMIN_SSO_REDIRECT_CALLBACK URL the SSO flow redirects to after browser login.

  • ADMIN_SSO_REDIRECT_SUCCESS URL the backend redirects to after successful SSO completion.

  • ADMIN_SSO_AUTH_METHOD OAuth client auth method: client_secret_basic (client secret) or private_key_jwt (client certificate).

  • ADMIN_SSO_AUTH_CREDENTIAL OAuth credential. Must be either an Entra ID client secret or a base64-encoded private key (for private_key_jwt). See How to generate Azure credentials

  • ADMIN_SSO_PUBLIC_CERT Base64-encoded public certificate. Required when ADMIN_SSO_AUTH_METHOD=private_key_jwt.

  • AZURE_SERVER_METADATA_URL Azure tenant OpenID Connect metadata URL. Example: https://login.microsoftonline.com/{tenant_id}/v2.0/.well-known/openid-configuration

Entra ID Graph API (beta)

LLMGW can be configured to use Graph API to connect to Entra ID to fetch user and group info. This can be used to import groups to admin portal or allow to fetch user information via user Entra object id. In order to enable this functionality App Registration need to be setup with following permissions:

  • Group.Read.All, GroupMember.Read.All, User.Read.All
  • or can be set to Directory.Read.All

Variables to configure

  • GRAPH_API_ENABLED Whether Graph API should be enabled for importing user/group info from Entra (required to support entra identity).

  • GRAPH_API_TENANT_ID App Registration Tenant ID.

  • GRAPH_API_CLIENT_ID App Registration Client ID.

  • GRAPH_API_CLIENT_SECRET App Registration Client Secret.

OIDC (DEPRECATED)

  • LLMGW_API_OIDC_ENABLED Whether OIDC should be enabled for authentication LLMGW requests.

  • LLMGW_API_OIDC_TENANT_ID Tenant Id used for OIDC flow.

  • LLMGW_API_OIDC_CLIENT_ID Client Id used for OIDC flow.

  • LLMGW_API_OIDC_CLIENT_SECRET Client secret used for OIDC flow.

Azure

These are only relevant when deploying on Azure.

  • AZURE_CLIENT_ID Client ID of the user-assigned managed identity used by the container.

  • APPLICATIONINSIGHTS_CONNECTION_STRING Connection string for Azure Application Insights.

Logging

  • LOG_FILE Optional path to a log file. If empty or not set, logs go to stdout only.

  • STD_JSON_LOG Output logs as JSON to stdout.

    • Default: false.

  • FILE_JSON_LOG Output logs as JSON to LOG_FILE (requires LOG_FILE to be set).

    • Default: false.

  • LOG_RESOURCE_USAGE Periodically log CPU and memory usage.

    • Default: false.

  • LOG_USER_DATA Log user request/response payloads. Caution: useful for debugging, but not recommended in production due to potential sensitive data exposure.

    • Default: false.

Generate private key and public certificate for SSO cert-based flow

This can be done in multiple ways. A bash instruction is shown below. Requires openssl and base64 utils.

  1. Generate a private key:

    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out app-private.pem && \
chmod 600 app-private.pem

  2. Generate a public certificate using that key:

    openssl req -new -x509 -key app-private.pem -sha256 -days 730 \
  -subj "/CN=mytestapp" -out app-public.cer

  3. Turn both to b64:

    base64 < app-public.cer | tr -d '\n' > app-public.cer.b64 && \
base64 < app-private.pem | tr -d '\n' > app-private.pem.b64

  4. Upload the public cert file to EntraID

  5. Make b64 secret available via ADMIN_SSO_AUTH_CREDENTIAL var.
  6. Make b64 certificate available via ADMIN_SSO_PUBLIC_CERT var.