Skip to content

LLMGW container

This page lists the environment variables supported by the LLMGW container. Variables are grouped by area and include defaults and usage notes where applicable.

App

  • DEBUG Whether the app is in debug mode. Mainly affects logging and telemetry.
    • Default is false.
  • ENVIRONMENT Deployment environment.
    • Supported values: dev, test, prd, local.
  • SESSION_SECRET_KEY The key used to protect FastAPI/Starlette session cookies. Should be at least 256 bits in production.
  • ADMIN_API_KEY Admin API key that can be used for access to the admin portal.
  • ALLOWED_HOSTS A comma-separated list of hosts against which the incoming request's Host header will be checked.
    • Example: ["localhost", "my-test-domain.net"]
  • CORS_ALLOWED_ORIGINS Browser origins allowed by the CORS middleware. Required for browser-based clients such as the Anthropic Claude in Office Add-in. Backend-to-backend integrations do not need an entry. Default is empty (no browser frontend allowed).
    • Example: ["https://pivot.claude.ai"]
  • ADMIN_PORTAL_BRAND_COLOR Brand color (hex color code for primary brand color).
  • ADMIN_PORTAL_BRAND_ICON Brand icon (base64-encoded PNG for favicon).
  • ADMIN_PORTAL_BRAND_LOGO Brand logo (base64-encoded PNG for sidebar header).

Configuration sources

  • CONFIG_SOURCES A JSON list of configuration source definitions. Always in a format of *prefix*:*service-specific source identifier*. Allowed prefices are dir, aws_s3 and azure_blob..
    All sources must use the same backend, e.g., s3: cannot be mixed with azureblob:. For Azure Blob Storage the format is <container>/<blob>; for AWS S3 <bucket>/<file_path>; for local directory a filepath.

Examples: 

[
  "dir:/etc/llmgw",
  "dir:./config"
]

[
  "s3:config-bucket/config-prod.yaml",
  "s3:config-bucket/extra-models.yaml"
]

Entities

  • ALLOW_ANONYMOUS_ENTITIES Allow requests without a defined entity (user/project).
    • Default: true (helpful when spend limits are not enforced).

Secrets and secret stores

  • SECRET_STORE Secret provider backend used to resolve the secrets.

    • Supported values: env, azure_kv, aws_secrets_manager.

  • APPLICATION_KEY_VAULT_URL Azure Key Vault URL used when SECRET_STORE=azure_kv.

    • Example: https://llmgw-kv.vault.azure.net/

  • ENV_SERVICE_CACHE_EXPIRATION_SECONDS Cache expiration time (seconds) for local secrets.

    • Default: 600

Storage backends

  • STORAGE Storage backend used to store/load the YAML config.

    • Supported values: dir, azure_blob, aws_s3.

  • AZURE_BLOB_STORAGE_ACCOUNT_URL Azure Blob Storage account URL (required when STORAGE=azure_blob).

Database

  • DB_CONNECTION_STRING The Postgres connection string in libpq (key=value) or URL style.

Interaction logs

These are only needed when interaction-log body storage uses the Azure Blob backend (interaction_log.body.storage.type: azure_blob in the configuration file). When set, all three are required: a missing or empty value fails on startup. They are read from env so the YAML config and the masked config API stay free of infra and secrets.

  • INTERACTION_LOG_AZURE_ACCOUNT_URL Azure Blob Storage account URL used to store request/response bodies.

    • Example: https://myaccount.blob.core.windows.net

  • AZURE_SUBSCRIPTION_ID Azure subscription ID. Used to manage the blob lifecycle (retention) rule.

  • AZURE_RESOURCE_GROUP_NAME Resource group that holds the storage account. Used to manage the blob lifecycle (retention) rule.

Required Azure permissions

The container's managed identity needs two role assignments on the storage account. They are split deliberately: writing log bodies is a data-plane operation scoped to one container, while managing the retention rule is a management-plane operation that Azure only exposes at the storage-account level.

  • Storage Blob Data Contributor — scoped to the container configured in interaction_log.body.container. Data-plane access for uploading the request/response bodies.

  • Storage Account Contributor — scoped to the whole storage account. Management-plane access so the app can create and update the blob lifecycle (retention) policy. Retention is enforced by an Azure blob lifecycle rule (named interaction-log-body-retention) that the app upserts at startup from interaction_log.body.retention_days. The rule deletes blobs under the interaction-logs/ prefix after the configured number of days. Azure lifecycle/management policies are an account-level resource, which is why this role is granted at account scope rather than per container.

This is why AZURE_SUBSCRIPTION_ID and AZURE_RESOURCE_GROUP_NAME are required alongside INTERACTION_LOG_AZURE_ACCOUNT_URL: the management-plane SDK call that maintains the retention rule needs the subscription and resource group to address the storage account.

OTEL

  • OTEL_SDK_DISABLED Disable OpenTelemetry configuration.

    • Default: false (OTEL enabled)

  • OTEL_TRACES_EXPORTER OTEL Traces exporter.

    • Supported values: otlp, console, or none.
    • Default: otlp.

  • OTEL_METRICS_EXPORTER OTEL Metrics exporter.

    • Supported values: otlp, console, or none.
    • Default: otlp.

  • OTEL_LOGS_EXPORTER OTEL Logs exporter.

    • Supported values: otlp or none.
    • Default: otlp.

  • OTEL_EXPORTER_ENDPOINT OTLP exporter endpoint. Required if any of OTEL_*_EXPORTER is otlp.

    • Example: http://<SERVICE_HOST>:4318
    • Azure Container Apps: <SERVICE_HOST> is the ACA service name (e.g., bss-llm-gateway-dev-otel).

  • OTEL_PERIODIC_EXPORT_INTERVAL_MILLIS OTEL periodic export interval (ms).

    • Default: 60000

  • DEBUG_METRICS When main exporter is otlp, enable additional console exporters / debug messages.

    • Boolean flag: 1, true, True.

Admin portal and SSO

SSO must be configured for admin portal access. Currently only Entra ID is supported. For other identity providers, please contact us.

  • ADMIN_AUTH_ENABLED Enable admin authentication/SSO. Warning: Disabling this removes SSO protection and auth endpoints.

  • ADMIN_SSO_CLIENT_ID OAuth client ID (e.g., Entra ID App ID).

  • ADMIN_SSO_REDIRECT_CALLBACK URL the SSO flow redirects to after browser login.

  • ADMIN_SSO_REDIRECT_SUCCESS URL the backend redirects to after successful SSO completion.

  • ADMIN_SSO_AUTH_METHOD OAuth client auth method: client_secret_basic (client secret) or private_key_jwt (client certificate).

  • ADMIN_SSO_AUTH_CREDENTIAL OAuth credential. Must be either an Entra ID client secret or a base64-encoded private key (for private_key_jwt). See How to generate Azure credentials

  • ADMIN_SSO_PUBLIC_CERT Base64-encoded public certificate. Required when ADMIN_SSO_AUTH_METHOD=private_key_jwt.

  • AZURE_SERVER_METADATA_URL Azure tenant OpenID Connect metadata URL. Example: https://login.microsoftonline.com/{tenant_id}/v2.0/.well-known/openid-configuration

Entra ID setup

To enable SSO via Entra ID, the following steps are required:

Note: When you create an App Registration, Azure automatically creates a corresponding Enterprise Application. Steps 1–4 are performed in the App Registration blade. Step 5 is performed in the Enterprise Application blade, which controls who can actually sign in and what role they receive.

  1. Create an App Registration in Azure Entra ID.
  2. In the App Registration, go to Authentication and add the redirect URI: https://{app-url}/api/admin/login/oauth2/azure/callback.
  3. In the App Registration, go to Certificates & secrets and generate a client secret or upload a certificate.
  4. In the App Registration, go to App roles and create the following roles: Admin, ProjectManager, Auditor, SuperAdmin, User.
  5. In the Enterprise Application, configure user access:
    1. Navigate to Azure Entra ID → Enterprise applications and find the application created in step 1.
    2. Under Properties, set Assignment required? to Yes to disable default access.
    3. Under Users and groups, click Add user/group and assign users or groups with the appropriate app role (Admin, ProjectManager, Auditor, SuperAdmin, or User).
  6. Set the environment variables listed above in your deployment.

For more information about RBAC and role permissions, see the Admin portal documentation.

Entra ID Graph API

LLMGW can be configured to use the Microsoft Graph API to fetch user and group information from Entra ID. This enables importing groups into the admin portal and looking up user details via the Entra user header.

To enable this functionality:

  1. Create an Entra ID App Registration.
  2. Create a client secret for the App Registration.
  3. Assign the following API permissions:
    • Group.Read.All, GroupMember.Read.All, User.Read.All
    • or alternatively Directory.Read.All

Variables

  • GRAPH_API_ENABLED Whether Graph API should be enabled for importing user/group info from Entra ID (required to support Entra identity).

  • GRAPH_API_TENANT_ID App Registration Tenant ID.

  • GRAPH_API_CLIENT_ID App Registration Client ID.

  • GRAPH_API_CLIENT_SECRET App Registration Client Secret.

  • GRAPH_API_GROUP_MEMBERSHIP Controls how user group memberships are resolved via Graph API. One of:

    • transitive (default) — flattened group membership including nested groups.
    • direct — only directly-assigned groups (prior behaviour).
    • none — skip the group fetch entirely; users are treated as having no groups.

    Note: this variable usually does not need changing. Change it from the default only after carefully evaluating the implications for user entitlements and group-based access.

  • GRAPH_API_GROUP_PAGE_SIZE Page size (Graph $top parameter) used when listing user groups. Defaults to 999, the documented maximum for directory objects. Results are paginated via @odata.nextLink until exhausted.

Agent overview

These configure the optional Agents Overview section of the admin portal, which proxies to the librechat-auditor service to display the LibreChat agents in a tenant.

Enabling the feature requires both of the following:

  1. The YAML config flag agent_overview.enabled: true (default false) — the rollout switch for the section.
  2. The transport environment variables below, pointing the gateway at the librechat-auditor service.

The transport variables are:

  • LIBRECHAT_AUDITOR_BASE_URL Base URL of the librechat-auditor service the gateway proxies agent requests to.

    • Default: empty (feature disabled).

  • LIBRECHAT_AUDITOR_API_KEY Shared API key the gateway attaches when calling the auditor service.

    • Default: empty (feature disabled).

  • LIBRECHAT_AUDITOR_HTTP_TIMEOUT_SECONDS Request timeout (seconds) for calls to the auditor service.

    • Default: 30.

If the base URL or API key is missing, the section stays disabled (the API returns 503) even when the flag is on.

Azure

These are only relevant when deploying on Azure.

  • AZURE_CLIENT_ID Client ID of the user-assigned managed identity used by the container.

  • APPLICATIONINSIGHTS_CONNECTION_STRING Connection string for Azure Application Insights.

Google

WIF Authentication

  • GOOGLE_VERTEXAI_PROJECT_ID - The ID of the project on behalf of which VertexAI requests are going to made. Used both for model call URL construction and Google WIF.
  • GOOGLE_WIF_CONFIG - A json containing the Google WIF config.
  • GOOGLE_EXTERNAL_ACCOUNT_ALLOW_EXECUTABLES - Must be set to 1 if the provided Google WIF Config uses an executable to obtain identity keys.
  • GOOGLE_ENTRAID_RESOURCE_URI - The API URL of the EntraID App Registration used to impersonate the Google Service Account.

See Google WIF Authentication setup for details on setting up the WIF Authentication.

Logging

  • LOG_FILE Optional path to a log file. If empty or not set, logs go to stdout only.

  • STD_JSON_LOG Output logs as JSON to stdout.

    • Default: false.

  • FILE_JSON_LOG Output logs as JSON to LOG_FILE (requires LOG_FILE to be set).

    • Default: false.

  • LOG_RESOURCE_USAGE Periodically log CPU and memory usage.

    • Default: false.

  • LOG_REQUEST_RESPONSE_BODY Log request and response bodies for successful proxy responses. Caution: useful for debugging, but not recommended in production due to potential sensitive data exposure. Regardless of this setting, response bodies are always logged for non-success responses (at WARNING) to aid incident investigation. Request/response headers and query parameters are only logged when the logger level is DEBUG.

    • Default: false.
    • Formerly named LOG_USER_DATA; the old name is still accepted as an alias but is deprecated and should be removed from deployment configuration. On the Terraform side the variable was renamed llmgw_log_user_data → llmgw_log_request_response_body with no alias, so deployment pipelines must be updated.

Alerting

  • ALERTMANAGER_URL Base URL of the Prometheus Alertmanager /api/v2/alerts endpoint that the gateway posts spend-limit alerts to. When unset or empty, alerting is disabled: no alerts are posted and the admin portal Alerts view returns empty lists. In production this is set by Terraform when enable_alertmanager = true; in local development use the docker-compose Alertmanager service.

    • Example: http://my-alertmgr/api/v2/alerts
    • Default: unset (alerting off).

  • ADMIN_PORTAL_URL Public URL of the admin portal frontend. Used as the generatorURL field on posted Alertmanager alerts so the Alertmanager UI can link back to the portal.

    • Default: http://localhost:5173.

  • SPEND_ALERT_RULES_CACHE_TTL_SECONDS How long (in seconds) the gateway caches the alert rules fetched from the database before re-querying. Increasing this reduces database load at the cost of slower propagation of rule changes.

    • Default: 60.

  • SPEND_ALERT_ENTITY_CACHE_TTL_SECONDS How long (in seconds) contact-email lookups for recipient token resolution are cached. Stale values are acceptable because contact information changes rarely.

    • Default: 300.

Generate private key and public certificate for SSO cert-based flow

This can be done in multiple ways. A bash instruction is shown below. Requires openssl and base64 utils.

  1. Generate a private key:

    openssl genpkey -algorithm RSA -pkeyopt rsa_keygen_bits:3072 -out app-private.pem && \
chmod 600 app-private.pem

  2. Generate a public certificate using that key:

    openssl req -new -x509 -key app-private.pem -sha256 -days 730 \
  -subj "/CN=mytestapp" -out app-public.cer

  3. Turn both to b64:

    base64 < app-public.cer | tr -d '\n' > app-public.cer.b64 && \
base64 < app-private.pem | tr -d '\n' > app-private.pem.b64

  4. Upload the public cert file to EntraID

  5. Make b64 secret available via ADMIN_SSO_AUTH_CREDENTIAL var.
  6. Make b64 certificate available via ADMIN_SSO_PUBLIC_CERT var.

Google WIF Authentication setup

In order to call VertexAI LLMGW has to obtain access to the platform. This can be done in various ways, but Google's Workload Identity Federation with Service Account Impersonation is the most secure as per Google's docs. More information can be found in the Workload Identity Federation (WIF) documentation.

WIF With Azure Container Apps

Related documentation: Workload Identity Federation with other clouds

Since the documentation is written for Azure VMs, the setup for Container APPs will be a little different:

  1. Setup GCP Project and Service Account as per docs.
  2. Create a Microsoft EntraID App Registration.
  3. Expose an API for App Registration. You should get an ID that looks like api://12a45b78-1c11-aa2a-b3b-123456789102. Set it as GOOGLE_ENTRAID_RESOURCE_URI env var.
  4. Configure WIF following the Google docs mentioned before.
  5. Authenticate the App Registration by following the Service Account Impersonation section in the docs.
  6. Create a json to place in the GOOGLE_WIF_CONFIG env var by executing this shell command (requires gcloud utility):
gcloud iam workload-identity-pools create-cred-config \                        
  projects/*PROJECT NUMBER*/locations/global/workloadIdentityPools/*POOL ID*/providers/*PROVIDER ID* \
  --service-account=*SERVICE ACCOUNT EMAIL OR NUMBER* \
  --service-account-token-lifetime-seconds=3600 \
  --output-file=*OUTPUT PATH TO JSON* \
  --executable-command="python /app/get_azure_entra_token_google_impersonation.py" \ # do not change this
  --executable-timeout-millis=15000

Note that the container app has to have a proper Managed Identity in order to authenticate against the created App Registration. If User-Assigned Identity is used, its ID passed in AZURE_CLIENT_ID var is going to be used for EntraID token negotiation.