Skip to main content
Give one model name several deployments, so calls keep working when one provider is slow or down. You set this up once on the model in Model Gateway studio; callers keep sending the same model name.

​
How a pooled call is routed

A model with a Deployment pool routes every call that names it across the pool’s deployments. Each deployment is another enabled model of the same modality in the same workspace.
  1. The gateway orders the deployments by weight, at random, without repeats. A deployment with weight 3 is picked first three times as often as one with weight 1.
  2. It skips deployments that are off, can’t serve the request, are outside the allowed regions, or are cooling down after a recent failure.
  3. It calls the first remaining deployment. If that call fails with a rate limit, overload, timeout, transport error, or 5xx, and no output has reached you yet, it tries the next deployment.
  4. It stops when a deployment answers, when it has made the maximum number of attempts, or when the request deadline passes.
Once a model has a pool, calls naming it go only to the pool’s deployments. A pooled model can’t be a deployment in another pool. The gateway does not retry an authentication failure, an invalid request, a content refusal, or a reply it can’t decode. Once a reply has started streaming to you, a failure ends the stream with an error instead of starting a second answer.

​
Recovery settings

Recovery applies to every model, with or without a pool. Without a pool, a retry goes back to the same deployment after its cooldown. A provider’s Retry-After is respected within the deadline. A fallback never resets the deadline.

​
Region restriction

Each deployment can name the region it serves from, such as us-east-1. A pool can list Allowed regions, and then routes only to deployments whose serving region is on that list. A deployment with no serving region is skipped by a pool that lists regions. A caller can narrow further, per request, with the x-gateway-model-regions header:
The header narrows and never widens. A call is served only from a region on both the pool’s list and the header. For a model without a pool, the header requires the model’s own serving region to be listed. The header takes up to 32 comma-separated region names, each 1 to 64 letters, digits, -, or _.

​
Before you begin

  • You’re an organization admin. Only organization admins can change a model’s settings.
  • The models you want as deployments are on, have the same modality, and live in the same workspace as the model you’re configuring.
1

Open the model's settings

In Model Gateway studio, open Models, select the model your callers name, and open the Settings tab.
2

Set each deployment's serving region

Skip this step if you don’t restrict regions. For each model you’ll add to the pool, open its Settings, find Serving region, choose Override, and enter the Region, such as us-east-1. Select Save changes.
3

Add the deployment pool

Back on the model your callers name, find Deployment pool and switch from Direct (default) to Pool. For each row, choose a Deployment and enter a Weight from 1 to 10,000. Select Add deployment for more rows, up to 16. To restrict regions, enter Allowed regions, comma-separated, such as us-east-1, us-west-2.
4

Adjust recovery

Optional. Under Recovery, switch from Default to Override and set Attempts, Deadline, ms, Backoff, ms with its maximum, and Cooldown, ms.
5

Save the model

Select Save changes. The bar above it lists what will change before you save.

​
Verify

  1. On the model’s Overview tab, Routing and health reads, for example, Weighted across 2 deployments, only in us-east-1. and Recovery is overridden: … when you changed recovery.
  2. Make a call that names the model. It answers as before; nothing changes for the caller.
  3. Open Calls and select the call. The drawer’s subtitle shows the number of attempts, and Served by shows the provider that answered.
If no deployment qualifies, for example when every deployment is outside the regions you asked for, the call returns 400 with No authorized deployment meets the route constraints.