Skip to content

Release Notes

v2.7.1 (10.7.2026)

Features

  • Model residency — see Residency configuration and LLM Providers.
    • Each Azure, AWS, and Google model is classified against a configured preferred residency.
    • New Providers view in the admin portal shows per-model residency labels and compliance badges.

v2.7.0 (1.7.2026)

Features

  • Interaction log management — the admin portal gains full control over interaction logs. See Interaction logs and interaction logs configuration.
    • Enable or disable interaction logging, sensitivity and request/response body storage per project, overriding the app-wide defaults.
    • Filter interaction logs in the admin portal.
    • Export interaction logs — metadata.csv plus a ZIP archive with the stored request/response bodies. Row and concurrency caps are configurable, and every export is recorded in the admin audit log (who, scope, row count).
    • Rows can be marked sensitive; their bodies are readable only by the Auditor role after an explicit "Reveal sensitive content" step.
    • Cursor pagination (next_cursor/prev_cursor) and a has_more flag on the audit-log and interaction-log endpoints.
  • Spend limit alerting — the gateway now integrates with Prometheus Alertmanager for spend notifications. See Alertmanager configuration and Alerts.
    • Fires a SpendLimitThresholdReached alert when an entity's spend crosses a configured cost_limit_alert threshold (e.g. 80% of limit); evaluation runs off the request path.
    • Configurable severity (info/warning/error/critical) per alert rule.
    • Alert recipients are resolved to email addresses; the <project>, <group>, and <user> tokens expand to the alerted entity's contact email (with sensible fallbacks). Delivery is via Alertmanager email.
    • Read-only Alerts view in the admin portal (new viewAlerts permission), filtering active-only vs. active-plus-suppressed alerts with a detail panel. Visible only when ALERTMANAGER_URL is set.
  • Azure Cohere reranker support — new /azure-cohere endpoint supporting Cohere's Reranker model. See Azure Cohere client.
  • Mistral models via OpenAI — limited support, Document AI use only.
  • Project contact email & Entra group access — projects now have a dedicated contact email (separate from the owner) and can grant access to Entra ID groups via a new additive grantees field, both settable on the create and edit forms in the admin portal. See Projects.
  • Configurable outbound HTTP timeout — PROXY_HTTP_TIMEOUT_SECONDS controls the timeout for outbound proxy calls. See LLMGW environment configuration.
  • Broader view switching — the admin portal's switch between personal and management views is now shown to any user holding both a personal-view role and a management role (e.g. Auditor+User, Admin+User); previously it appeared only for SuperAdmin and ProjectManager.
  • Admin portal redesign — tabbed alerts and logs, and refactored project detail fields.
  • Agents overview (preview) — a feature-flagged admin-portal view listing LibreChat agents (status, model, author, tools), read on demand by proxying to the LibreChat auditor service with no data persisted in llmgw. Off by default; enable via agent_overview.enabled. See Agents and Agent overview configuration.

Fixes

  • Streaming token usage for Azure-Anthropic via the OpenAI route — streaming responses now include a final chunk carrying the full token usage, matching non-streaming responses and the AWS Bedrock path.
  • Azure api-version preserved on legacy /openai/... requests — the parameter is no longer dropped from proxied requests (which had started returning 404 "Resource not found"); inbound query params are now merged onto the target URL explicitly (httpx bumped to ^0.28.1).
  • Resolved May–June CVEs.

Upgrade Notes

  • New/updated settings — see LLMGW environment configuration:
    • ALERTMANAGER_URL — enables alerting and the Alerts view; alerting is a no-op when unset.
    • SPEND_ALERT_RULES_CACHE_TTL_SECONDS — alert rule cache TTL (default 60s).
    • PROXY_HTTP_TIMEOUT_SECONDS — default raised from 120s to 240s.

v2.6.0 (15.5.2026)

Features

  • Admin portal home page — New home screen with gateway-wide KPIs: spend trend, request volume, top projects by spend, top users. Each widget has a sparkline and period-over-period trend indicator. See Home.
  • Project usage charts — New Usage tab in project detail view: cumulative spend, remaining budget, spend per model. See Projects.
  • SuperAdmin role — New SuperAdmin Entra ID app role with full cross-tenant permissions. Add the role to your Entra ID App Registration and assign users/groups in the Enterprise Application. See Admin portal documentation and Entra ID setup.
  • AWS secrets health check — Dedicated health check for AWS Secrets Manager connectivity.
  • Support for Anthropic Claude in Office Add-in — The add-in can be configured to use LLMGW as gateway. See Anthropic Claude in Office Add-in section for more details.
  • Support for gpt5.4/gpt5.5+ pricing — Support for premium (over 272k tokens) pricing model of gpt5.4/gpt5.5+ models. See cost_profiles section in Models Configuration.

Fixes

  • UTF-8 streaming parse error — Fixed crash when a multi-byte UTF-8 character was split across HTTP chunk boundaries in streaming responses.
  • OpenAI/Anthropic transformation fixes — Improved tool-use transformations, data blocks, and redacted thinking blocks when translating between OpenAI and Anthropic message formats.
  • Azure Key Vault secrets retrieval — Fixed intermittent failures fetching secrets from Key Vault.
  • Token revocation alert — Clearer message shown on token revocation.
  • Token creation error message — Specific error surfaced on token creation failure instead of a generic one.
  • Async call duration measurement — Call duration now measured correctly for async proxy targets.
  • Smoke test preflight — Pre-flight checks now correctly use the ENVIRONMENT variable.

Upgrade Notes

  • Admin portal application roles - Add User and SuperAdmin App roles to App Registration of Admin Portal and assign users and groups to the new roles in the Enterprise Application. See Entra ID setup.
  • PostgreSQL max_connections — It's recommended to set max_connections to at least 200 on your PostgreSQL instance. The right value depends on your deployment size and concurrency requirements; adjust upward for high-traffic deployments.

v2.5.1 (27.4.2026)

Breaking Changes ⚠️

  • LOG_USER_DATA renamed to LOG_REQUEST_RESPONSE_BODY
    • The old name is still accepted at runtime via an alias but is deprecated — update deployment configuration.
    • The Terraform variable was renamed in lockstep: llmgw_log_user_data → llmgw_log_request_response_body. No alias is provided on the Terraform side — pipelines referencing TF_VAR_llmgw_log_user_data must be updated.
    • Production deployments now default this value to false (request/response bodies are no longer logged in production by default).
    • See LLMGW environment configuration.
  • Default Entra ID group lookup is now transitive — EntraIdIdentityService now calls transitiveMemberOf with pagination by default, so authenticated users will generally see more groups than before. Any entitlement or authorisation logic keyed on the exact group list should be reviewed. To preserve prior behaviour, set GRAPH_API_GROUP_MEMBERSHIP=direct.
  • Azure Terraform provider versions tightened — azurerm raised to ~> 4.0 and azuread raised to ~> 3.0. Existing workspaces must run terraform init -upgrade before the next plan. Lock files (.terraform.lock.hcl) are now committed for core, models, and shared.

Fixes

  • Complete Entra ID group membership resolution — user groups are now fetched via the transitive Graph API endpoint and paginated through all pages. Two prior bugs are fixed:
    • Users inheriting entitlements only via a parent/nested group now receive those groups.
    • Users belonging to more than ~100 groups are no longer silently truncated.
  • Behaviour is configurable via two new env vars, GRAPH_API_GROUP_MEMBERSHIP (defaults to transitive) and GRAPH_API_GROUP_PAGE_SIZE (defaults to 999). Set GRAPH_API_GROUP_MEMBERSHIP=direct to retain the previous direct-only semantics. See LLMGW environment configuration.
  • No new Graph API permissions are required — GroupMember.Read.All covers both the transitive and direct endpoints.
  • VertexAI OpenAI-compatible target — leading /v1/ is now stripped from the client path before forwarding, matching what the Vertex AI OpenAI-compatible endpoint expects.

Improvements

  • Logging behaviour
    • Successful proxy responses now log at INFO; non-success responses log at WARNING. Log-based alerts keyed on log level may need updating.
    • Response bodies are always logged on non-success status regardless of LOG_REQUEST_RESPONSE_BODY, to aid incident investigation.
    • Request/response headers and query parameters are only included at DEBUG.
    • Raw payloads removed from exception log extras in the OpenAI SSE and Azure Responses parsers to avoid leaking sensitive content into error logs.

v2.5.0 (30.3.2026)

Features

  • Support for Gemini models via OpenAI
  • Support for STT/TTS
  • Support for Anthropic models (AWS Bedrock and Azure)
    • Claude models can now be accessed via both AWS Bedrock and Azure providers through the Anthropic endpoint
    • see Anthropic setup
  • RBAC support for admin portal
  • Support for LLM provider configuration in Admin portal

Fixes

  • Admin portal UI is shown before login

Breaking Changes ⚠️

  • Roles must be configured for the admin portal app registration
    • Roles must be configured as per here
  • Removed support for OIDC (preview feature)
    • OpenID token authentication was removed from roadmap and won't be supported
  • Replaced llmgw-user-openid-id header with llmgw-entra-user

v2.4.0 (27.2.2026)

Breaking Changes ⚠️

  • Admin API key can be used only for admin/stats/llm-stats endpoints (billing info), rest of admin endpoints require auth via SSO.

  • Removed support for API keys in config

  • Only API keys created via admin portal are supported
  • If required to migrate existing keys from config you can do it directly in database, see sample query below
INSERT INTO llmgw_entities (name, entity_type, owner, display_name)
VALUES ('test_app','project','user@adastragrp.com','test_app')
-- returns entity_id
INSERT INTO llmgw_api_keys (name, key_reference, created_by, created_at,  expires_at, valid_for, entity_id) 
VALUES 
('test_app_key','azure:https://bss-llm-gateway-test.vault.azure.net/secrets/test-app-key','user@adastragrp.com','2025-02-26T00:00:00','2026-02-26T00:00:00', 365, [entity_id]);

Features

  • Full Entra ID integration (beta)

    • Ability to import groups in Admin portal
    • Ability to authenticate via user Entra ID OID and fetch users groups
    • To configure you must set the required env vars
    • For more information on configuring Entra and Graph API access, see the LLMGW environment configuration.

  • Enhanced token pricing support

    • Full support for cached tokens pricing for Azure, AWS, OpenAI LLMs
    • Correct support for image generation tokens pricing
    • New format for pricing tokens, see more in pricing configuration
    • Improvements to tiktokenizer count based on LLM provider

  • Admin portal improvements

    • Limit model access per project
    • Disallow editing default spend limits when group spend limit is set
    • Support for importing Entra ID groups
    • Better warning message when deleting project with API keys
    • Allow deletion of expired API keys

  • Coding assistants integration

    • Added openai/models and openai/v1/models endpoints to support Xcode Coding Intelligence and Android Studio custom model integration. For details see OpenAI client support

v2.3.1 (3.2.2026)

Bugfixes

  • Fix nginx config setup for the admin portal (frontend docker image)

v2.3.0 (26.1.2026)

Breaking Changes ⚠️

  • Remove support for service bus integration - Configuration is no longer propagated from storage via service bus. If you update the configuration, the container must be restarted.

  • LLMGW environment variables type change - The following variables were changed from *_NAME style to concrete values:

    • SESSION_SECRET_KEY_NAME → SESSION_SECRET_KEY
    • ADMIN_API_KEY_NAME → ADMIN_API_KEY
    • DB_CONNECTION_STRING_SECRET_NAME → DB_CONNECTION_STRING
    • ADMIN_SSO_AUTH_CREDENTIAL_NAME → ADMIN_SSO_AUTH_CREDENTIAL
    • LLMGW_API_OIDC_CLIENT_SECRET_NAME → LLMGW_API_OIDC_CLIENT_SECRET

    These will no longer be retrieved dynamically from a secrets manager by the application code, but should be provided at deployment time.

  • New/updated variables - see envconfig

    • ADMIN_PORTAL_ variables to configure admin portal
    • CONFIG_SOURCES, STORAGE and AZURE_BLOB_STORAGE_ACCOUNT_URL - to configure config files location, please review environment variable documentation for updates
    • SECRET_STORE to configure secret store

  • Spend limits per group - When a spend limit is configured for a group in the Admin portal, only those groups with configured limits have access and no default limit applies.

Features

  • AWS deployment support - LLMGW can now be deployed with AWS services (documentation in progress)
  • Native Bedrock models - Use AwsBedrockNative models for native Bedrock integration (e.g., via Claude Code)
  • Entra ID integration (Phase 1)
    • Requests can now be authorized with Entra ID SSO token
    • Endpoints to retrieve Entra ID available groups
  • Enhanced spend limits - Project access can be limited to specific groups. Default user limit only applies if no project-user limit is set.
  • Admin portal improvements
    • Configure Admin portal branding (logo, colors)
    • Alphabetic ordering in filters for API keys and user-based tokens screens.
    • Better UX for spend limits
    • Better details in audit log (shows name of deleted entity)
  • Improved security - New NGINX security headers and app-based host header validation. Set ALLOWED_HOSTS for LLMGW container and ENABLE_HSTS for Frontend container. See LLMGW env vars reference and Frontend env vars reference for details.
  • Flexible NGINX configuration - NGINX now accepts additional PROXY_PORT and BACKEND_PROXY_PROTO environment variables

Improvements

  • OpenAI /responses API fixes
    • Support for reasoning responses/tokens
    • Support for custom named models on backend
  • Better error messages - Improved error messages for invalid request entities or misconfigured spend limits

Grafana

  • Dashboard versioning - Better version management for Grafana dashboards
  • Report data setup - Proper versioning for Dockerfile.reporting-data-setup following the LLMGW versioning
  • Cached tokens - Dashboards to display cached tokens used by LLM and approximate spend savings

v2.2.0 (10.11.2025)

Features

  • Responses API add support for OpenAI /responses endpoint. See details in OpenAI client
  • Specific limit priority — a specific limit (e.g. user limit on a project) always takes precedence over the default entity limit
  • Otel metric emitter config allows configuring how often metrics are pushed to OTEL via OTEL_PERIODIC_EXPORT_INTERVAL_MILLIS

Bug fixes

  • Admin portal fixes show correct number of active tokens, better default ordering
  • Azure key vault handle closing ssl connections properly

Grafana

  • Fixes and improvements to Grafana dashboards
  • NOTE: in order to fix data aggregation for projects reporting-data-setup latest image must be pulled and run again

v2.0.5 (19.10.2025)

Features

  • Grafana views container job template to create psql views for reporting

Improvements

  • Key vault caching better caching for key vault secrets
  • Admin portal improvements logout, sorting, currency indicator
  • Logging improvements fix exporting logs to otel, simplify logging

v2.0.1

Features

  • New database/backend model for managing entities, API keys, and user-based tokens
  • New spend entity model enabling spend limit management across entities
  • Admin portal UI for managing entities, spend limits, API keys, and user tokens
  • SSO support for Admin Portal endpoints (see documentation for environment variables)
  • AWS Bedrock support via boto3 integration
  • New embedding model - Added LLModelType (AzureOpenAIEmbedding) with cost computation support

Improvements

  • Enhanced Grafana setup
  • Streamlined configuration and settings
  • Improved Docker setup
  • Refactored token model

Breaking Changes

⚠️ Migration Required

  • ENVIRONMENT variable is now mandatory - Allowed values: local, dev, test, prd
  • Removed ALLOW_NON_CONFIGURED_ENTITIES - Boolean variable no longer supported
  • Spend limits configuration - No longer configurable via configuration file; manual migration required
  • Default entity types - Must be defined in configuration file
  • API keys deprecation - Configuration file support deprecated; manual migration to new model required
  • Ensure you configure the Key Vault secret cache time via AZURE_ENV_SERVICE_CACHE_EXPIRATION_SECONDS for expected admin portal behaviour

v1.7.5 (18.08.2025)

Features

  • Add support for EntraID token authentication

Bug fixes

  • Remove model from Custom Header Mapping

v1.7.4 (06.08.2025)

  • Refactoring auth layer
  • Add index for llmgw_spend_metrics to improve Grafana performance

v1.7.3 (31.07.2025)

  • new endpoints for managing entities configuration
  • remove support for id regex and soft spend limit in configuration
  • scripts for generation openapi spec

v1.7.2 (22.07.2025)

  • Security fix: SQLite version bump

v1.7.1 (22.07.2025)

  • More model graphs in Grafana
  • Add project daily usage to Grafana
  • Fix tracking response codes for errors

v1.7.0 (09.07.2025)

  • Add support for user based tokens

v1.6.7 (07.07.2025)

  • Better config validation
  • Retain spend metrics longer
  • Fix token usage count for azure models

v1.6.5 (02.07.2025)

  • Automatic cert refresh
  • Support for project/user API keys

v1.6.4 (23.06.2025)

  • Fix keyvault caching
  • Fix claude streaming
  • Fix token usage count for Bedrock models

v1.6.2 (19.06.2025)

  • Split and improve Grafana dashboards

v1.6.0 (31.03.2025)

  • Added support for alternative names for spend entities.
  • Added regex support for validating spend entity names.
  • Filtered out NonConfiguredSpendEntity objects from the database metadata.

v1.5.1 (31.03.2025)

  • All user spend entities can now have a default spend limit
  • Multiple spend entities are accepted in the request, e.g., llmgw-group=group_1,group_2

v1.5.0 (31.03.2025)

  • AWS Bedrock Claude support
  • Text to speech model support

v1.4.0 (07.03.2025)

  • Use loguru for logging.
  • Better model response status handling.
  • Unify currencies to Dollars.
  • Docs update.
  • Grafana dashboard update.
  • Add checking both HTTP headers and query parameters for spend entities.
  • DeepSeek model support.
  • AWS Bedrock support.

v1.3.0 (22.12.2024)

  • OTEL config update.
  • Grafana Dashboard tuning.
  • User logging_utils everywhere.
  • Poetry update.
  • Grafana SSO login and provisioning update.
  • Spend metrics update.
  • Gpt4 version update.
  • New llmgw headers and dynamic spend entities.
  • Flow-id propagation.

v1.2.0 (26.11.2024)

  • Metrics updates and fixes.
  • Grafana dashboards.
  • Load balancing based on rewards.
  • Non-text models support.
  • Provisioned Throughput Units (PTU) support.

v1.1.2 (29.10.2024)

  • Bug fixing.
  • Metrics refactoring.

v1.1.1 (24.10.2024)

  • Smoke test update.

v1.1.0 (24.10.2024)

  • Response time modeling experiments.
  • Bug fixing.
  • Streaming metrics.
  • Deployment pipeline update.
  • Doc update.
  • Model Heartbeat.
  • Spend limit metrics.
  • Docker image vulnerability scan.

v1.0.0 (14.10.2024)

Initial release of the LLM Gateway (LLMGW).