On this page
Inference
Reliability
Retries, failover chains, and what happens to billing when the upstream misbehaves.
Same-model retries
RelayRouter makes up to 2 attempts on retryable conditions: 429, 5xx, transport errors, and timeouts. Backoff honors the upstream Retry-After header, clamped to 200 ms–2 s; otherwise it uses roughly 500/900 ms plus jitter. Same-model retries happen even when failover is disabled.
Failover
Failover is enabled when the account setting auto_fallback is on, the key allows fallback, and the request does not set no_fallback. The relay walks either the per-request fallback_models array or the model's configured fallback_to chain.
- At most 4 models total are tried: the initial model plus up to 3 fallback hops.
- A hop the caller is not allowed to use is skipped.
- Failover is not attempted for no_api_key, provider_auth_error, or model_unavailable.
- Transport errors, timeouts, 429, 5xx, and empty responses are retryable and may fail over.
Billing during failover
The model that actually served the request is billed at its own price. A fallback is never billed at the cheaper requested model's rate. Failed upstream calls are never billed.
Timeouts and sanitized errors
The total request timeout is 300 seconds. Upstream errors are sanitized: 429 stays 429, while other upstream failures become 503 with a stable code. Cloudflare replaces raw 502/504 bodies, so clients should rely on the code field rather than upstream text.