API7 AI Gateway Token Rate Limit Configuration (Delivery)

API7 AI Gateway Token Rate Limit Configuration (Delivery)

1. 01

   Navigate to the AI Gateway section in the API7 Cloud console and select the target Gateway Group.

   Ensure the Gateway Group is in a region that supports the AI gateway features, such as Singapore, Frankfurt, or Virginia.

2. 02

   Create a new route for the LLM endpoint, specifying the upstream model provider and the model name.

   Use the route configuration to define the path and methods for the LLM API calls.

3. 03

   Enable the token rate limiting plugin on the route.

   In the plugin configuration, set the token budget and the time window (e.g., 100,000 tokens per minute) based on the client's usage pattern.

4. 04

   Configure the rate limit action to either reject or queue requests that exceed the token budget.

   Choose 'reject' to return a 429 status code, or 'queue' to hold requests until tokens are available, depending on the client's tolerance for latency.

5. 05

   Set up a budget alert in the AI Gateway settings to notify the client when token usage approaches the limit.

   Use the budget and rate limit features to define a threshold (e.g., 80% of the monthly token allocation) and integrate with email or webhook notifications.

6. 06

   Test the configuration by sending a burst of LLM requests to the route and verifying that the rate limiter triggers correctly.

   Use the API7 developer portal or a tool like curl to simulate traffic and check the response headers for rate limit information.

7. 07

   Document the token rate limit configuration in the client handoff guide, including the exact settings and how to adjust them.

   Provide the client team with instructions on how to modify the token budget in the API7 console and how to interpret the monitoring dashboards.