Spend Limits

Spend Limits

Spend limits in LLMGW help control resource usage by allowing administrators to define maximum usage amounts for different entities, either globally or with fine-grained overrides.

Spend Limit Hierarchy

Spend limits can be set at multiple levels, each taking precedence as follows:

1. Default Limit (Config File):

   • Each entity type (e.g. project, user, group) can have a default spend limit configured.
   • Only one duration (DAY, WEEK, MONTH) per entity type.
   • This serves as a fallback limit for any entity not explicitly set in the database.

2. Entity-Specific Limits (Database):

   • Can be configured via admin portal
   • More granular spend limits may be set per entity, for example:
     • Overall project limit, user group limit etc. (as per allowed entity types)
     • Per user or per user group limit within a project
     • Per user limit with user group
   • A single entity can have multiple spend limits (e.g. both per-user and per-group within the same project).
   • Different time periods can be defined (daily, weekly, monthly) for each limit.
   • Limit can be set for specific request identity (e.g. per specific user/user-group in given project)
   • Aliases (human-readable names) for entities are stored here.

Limit Application Logic

When a request is made, the system determines the relevant spend limit in this order:

1. Gather all limits from the database for entity connected with request (e.g project for api-key, user-based token)
2. For each request identity apply precedence:
   • Use the most-specific matching limit available.
   • If multiple applicable limits exist (e.g. both a user and a user group limit in the same project), the lowest limit wins.
   • If no matching DB limit is found, use the default for the entity type in the config file.

Example

Suppose the following limits are set for project agate:

• Project: $100/day
• User in project: $5/day
• User group “alpha” in project: $20/day
• User group “beta” in project: $10/day

A user in “alpha” making a request for “agate” has the following candidates:

• Project limit: $100
• Their user-group limit: $20
• Their personal limit: $5

→ Outcome: The lowest relevant limit applies, so their effective spend limit is $5/day.

Summary Table