Alertmanager

Prometheus Alertmanager is the gateway's alerting backbone. LLMGW POSTs alerts to it; Alertmanager owns grouping, deduplication, silencing, and routing to receivers (email, Slack, etc.).

Alertmanager is opt-in. Set ALERTMANAGER_URL on the LLMGW container to enable it. When the variable is absent, alerting is disabled and LLMGW starts cleanly with no dependency on Alertmanager. See LLMGW env variables for the full list of LLMGW-side variables that control alerting behaviour.

Connectivity

Direction	Protocol	Default port	Notes
LLMGW → Alertmanager	HTTP	9093	Used for posting alerts (POST /api/v2/alerts) and reading them back for the admin portal Alerts view (GET /api/v2/alerts). Alertmanager does not need to be reachable from the public internet — internal network access from LLMGW is sufficient.
Alertmanager → SMTP server	SMTP	587 (or as configured)	Required only when an email receiver is configured. Alertmanager initiates the connection to the relay; the relay does not need to reach Alertmanager.

Configuration

Alertmanager is configured via a single alertmanager.yml file, passed to the process with --config.file. The file controls global SMTP settings, the routing tree, and the list of receivers.

Example configuration

The example below covers the routing used for LLMGW alerts. Adapt SMTP settings and receiver addresses to your environment.

global:
  smtp_smarthost: 'smtp.example.com:587'
  smtp_from: 'alerts@example.com'
  smtp_auth_username: 'alerts@example.com'
  smtp_auth_password_file: /etc/alertmanager-secrets/smtp-password
  smtp_require_tls: true

route:
  receiver: 'ops-email'
  group_by: ['alertname', 'entity_type', 'entity_name', 'threshold_percent', 'reset_period']
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1h
  routes:
    - matchers:
        - alertname="SpendLimitThresholdReached"
        - recipients!=""
      receiver: 'finops-recipients'
    - matchers:
        - alertname="SpendLimitThresholdReached"
      receiver: 'finops-fallback'

receivers:
  - name: 'ops-email'
    # Default catch-all.
    email_configs:
      - to: 'ops@example.com'
        send_resolved: true

  - name: 'finops-recipients'
    email_configs:
      - to: '{{ .CommonLabels.recipients }}'
        send_resolved: true

  - name: 'finops-fallback'
    email_configs:
      - to: 'finops-team@example.com'
        send_resolved: true

Routing explained

The default routing tree handles the currently implemented alert types. New types can be added by inserting additional routes entries and receivers without any change to LLMGW itself.

Spend limit alert routing — alerts with alertname=SpendLimitThresholdReached are routed through two dedicated receivers:

Receiver	Matched when	Sends to
finops-recipients	Alert carries a non-empty recipients label	Comma-joined addresses in recipients, resolved by LLMGW at alert-build time from the entity's contact information and the rule's recipient configuration
finops-fallback	No recipients label present	A fixed fallback address (e.g. a shared FinOps mailbox)

Alerts that do not match any specific route fall through to the ops-email catch-all receiver.

SMTP password

Storing the SMTP password in the config file as plaintext is not recommended. Alertmanager supports smtp_auth_password_file to read the password from a file at runtime, which works well with secrets mounted into the container (e.g. from a secrets manager or a mounted secret volume).