Skip to content

Spend Limits

Spend limits in LLMGW help control resource usage by allowing administrators to define maximum usage amounts for different entities, either globally or with fine-grained overrides.

An entity is anything a limit can be attached to, such as a project, a user, or a user group. See Entities for more.

Spend Limit Hierarchy

Spend limits come from two places: config defaults and the database. A database limit replaces the config default for the same entity type.

  1. Default Limit (Config File):

    • Each entity type (e.g. project, user, group) can have a default spend limit configured.
    • Only one duration (DAY, WEEK, MONTH) per entity type.
    • This serves as a fallback limit for any entity not explicitly set in the database.

  2. Entity-Specific Limits (Database):

    • Set via the admin portal and stored in the database.
    • A project's configuration bundles two limits:
      • a project-wide limit, which caps the total spend of the project, and
      • a per-user limit, which gives each user in the project their own separate cap.
    • Per-group limits can also be added to a project. Each matched group gets its own cap. Group limits also restrict access: only members of a configured group can use the project. See Group Spend Limits.
    • Each limit can use its own time period (daily, weekly, monthly).
    • Human-readable aliases for entities are also stored here.

Note: Per-token limits (a unique cap for a single user, applied through their token) are upcoming and not yet available in the admin portal.

Spend limit alerts: When a spend limit is approaching its cap, the gateway can fire configurable alerts before the hard limit is actually hit. See Alerts for how to configure alert rules and view firing alerts.

Limit Application Logic

When a request is made, the system determines the relevant spend limit in this order:

  1. Gather all limits from the database for the entity connected with the request (e.g. project for api-key, user-based token).
  2. For each entity type, pick one limit:
    • if the database has limits for that type, the lowest matching one wins (e.g. when a user belongs to several groups);
    • otherwise the config default is used.
  3. Several entity types can apply to one request (e.g. project, user, and group). All their limits are enforced together, and the lowest of them is the effective cap.

Example

Suppose the following limits are set for project agate:

  • Project: $100/day
  • User in project: $5/day
  • User group "alpha" in project: $20/day
  • User group "beta" in project: $10/day

A user in "alpha" making a request for "agate" has the following candidates:

  • Project limit: $100
  • Their user-group limit: $20
  • Their personal limit: $5

→ Outcome: The lowest relevant limit applies, so their effective spend limit is $5/day.

Summary Table

Level Where Defined Example Scope Duration When it applies
Default entity type limit Config file All projects, all users, etc. One (day/week/month) When no DB limit is set for the type
Entity-specific limit Database Per project, per user, per group Any (day/week/month) Replaces the default for its type