Skip to content

Spend Limits

Spend Limits

Spend limits in LLMGW help control resource usage by allowing administrators to define maximum usage amounts for different entities, either globally or with fine-grained overrides.

Spend Limit Hierarchy

Spend limits can be set at multiple levels, each taking precedence as follows:

  1. Default Limit (Config File):

    • Each entity type (e.g. project, user, group) can have a default spend limit configured.
    • Only one duration (DAY, WEEK, MONTH) per entity type.
    • This serves as a fallback limit for any entity not explicitly set in the database.

  2. Entity-Specific Limits (Database):

    • Can be configured via admin portal
    • More granular spend limits may be set per entity, for example:
      • Overall project limit, user group limit etc. (as per allowed entity types)
      • Per user or per user group limit within a project
      • Per user limit with user group
    • A single entity can have multiple spend limits (e.g. both per-user and per-group within the same project).
    • Different time periods can be defined (daily, weekly, monthly) for each limit.
    • Limit can be set for specific request identity (e.g. per specific user/user-group in given project)
    • Aliases (human-readable names) for entities are stored here.

Limit Application Logic

When a request is made, the system determines the relevant spend limit in this order:

  1. Gather all limits from the database for entity connected with request (e.g project for api-key, user-based token)
  2. For each request identity apply precedence:
    • Use the most-specific matching limit available.
    • If multiple applicable limits exist (e.g. both a user and a user group limit in the same project), the lowest limit wins.
    • If no matching DB limit is found, use the default for the entity type in the config file.

Example

Suppose the following limits are set for project agate:

  • Project: $100/day
  • User in project: $5/day
  • User group "alpha" in project: $20/day
  • User group "beta" in project: $10/day

A user in "alpha" making a request for "agate" has the following candidates:

  • Project limit: $100
  • Their user-group limit: $20
  • Their personal limit: $5

→ Outcome: The lowest relevant limit applies, so their effective spend limit is $5/day.

Summary Table

Level Where Defined Example Scope Duration Precedence
Default entity type limit Config file All projects, all users, etc. One (day/week/month) Lowest, as fallback
Entity-specific limit Database Per project, per user, per group Any (day/week/month) Overrides default