Routing
One inbound request becomes an ordered list of attempts. This page is the mechanism, end to end: which route matches, which candidates it produces, how each candidate expands into legs, which failures retry the same upstream, which move on, which stop everything, and exactly what the attempt budget counts.
Note
Everything below is the behavior of the current build, which is private and pre-launch: the router prints an experimental notice on every start, and the config shape can still change. Where a key is accepted but does not yet do anything, this page says so.
The pipeline, in order
Five stages, always in this order. If you can name the stage a request died in, you have already narrowed the cause to one config block.
-
Route selection. The first route in declaration order whose
matchadmits the request wins. No match is a404. -
Candidate chain.
[model, ...models[]]when the request body carries a non-emptymodels[]; otherwise the route'schain, verbatim. -
Leg expansion. Each candidate becomes one leg per upstream that can
serve it, filtered and ordered by
provider{}. - The loop. Per leg: reserve budget, dispatch, classify the outcome, settle the reservation. Then the policy says commit, retry the same leg, advance, or surface the error.
- Commit. The first successful leg wins and the engine stops. Nothing re-routes after that.
Route selection
Routes are declared as an object keyed by route name, and matching walks them in
declaration order. The first admitting route wins, so put your specific routes above your
catch-all. Two keys can appear in match, and they do not work the same way.
An exact list of request paths, tested by membership: not a glob,
not a prefix. "/v1/chat/completions" does not admit
/api/v1/chat/completions; list both if you serve both surfaces.
Globs. Each entry is anchored and * becomes
.*, so "gpt-*" admits gpt-4o. Tested against
the request's model field.
An omitted key matches everything, so "match": {} is a catch-all. When no
route admits the request the client gets HTTP 404 with type
no_route_configured and the message
no route configured for path "<path>" and model "<model>".
Warning
surface is exact-match and models is glob. It is the easiest
pair in the config to get backwards, and getting it backwards looks like a routing
outage: every request 404s with no_route_configured.
The candidate chain
A candidate is a model slug the router is willing to try for this request. There are exactly two sources, and only one of them is the request body.
// request body carries a non-empty models[] …
candidates = [req.model, ...req.models]
// … otherwise the route's own chain is used, verbatim
candidates = route.chain
Warning
Without models[] in the request body, the request's
model field does not select a candidate at all. It feeds
route matching and the report's requested_model, and nothing else. A route
with "chain": ["gpt-4o"] and "match": {} answers a request for
{"model":"nope"} with 200 off the gpt-4o leg,
verified on this build. If you want the caller to steer the chain, the caller has to
send models[].
The candidate list is de-duplicated preserving first-seen order, so a slug repeated
between model and models[] is tried once, in its earliest
position.
Leg expansion
A leg is one concrete attempt: a candidate slug paired with one upstream and the
model id that upstream knows it by. Expansion walks the candidates in chain order, and for
each one the rule depends on whether the slug is declared in route.models.
-
Mapped slug (present in
route.models): one leg per declared upstream, and only those. An upstream'sunlistedModelsrule is ignored for a slug that has an explicit mapping. -
Unlisted slug: every upstream gets a chance to opt in via its own
unlistedModelsrule, in upstream declaration order:"verbatim"passes the slug through unchanged,"strip-author"drops everything up to and including the first/(openai/gpt-4obecomesgpt-4o), and"none"or unset means that upstream does not serve unlisted slugs.
A candidate that maps nowhere contributes zero legs and is skipped
silently, with no error and no leg entry in the report. If the whole chain maps
nowhere, the client gets HTTP 404 with
no upstream mapping for candidates: <list>.
provider{} composition: a request narrows, never widens
Both the route and the request body may carry a provider{} block. They are
combined per key, and the combination is deliberately one-directional: a caller can
restrict the upstreams their request touches, but cannot reach an upstream the route did
not already allow.
| Key | Config × request |
|---|---|
| order |
request order replaces config
order when present
|
| only | intersection of both lists |
| ignore | union of both lists |
| allow_fallbacks | any explicit false wins; default true |
| sort | request value, else config value, else none (and see the warning below) |
allow_fallbacks: false truncates the chain to the
primary candidate only, but all of that candidate's legs are still tried.
It bounds which models you may be served, not how many upstreams are attempted for the
model you asked for.
Request-side provider{} is coerced defensively: only string arrays survive,
and sort must be exactly "price" or "latency".
Anything else is discarded rather than honored.
Leg order comes from declaration order
Within a candidate, legs are ordered by three passes, in this fixed sequence:
-
provider.orderfront-loads the upstreams it names, in the order given; the rest stay behind them. onlyandignorefilter what survived.sortruns last, and is inert (see below).
So the baseline order is the order you wrote things down: for a mapped slug, the order of
upstream keys inside that slug's entry in route.models; for an unlisted slug,
the order upstreams are declared in route.upstreams. The only lever that
reorders legs today is provider.order.
Note
provider.sort orders a candidate's legs. "price" uses the
catalog price — cheapest first, with an unpriced leg sorting last rather than guessed
at. "latency" uses the rolling mean TTFT this process has observed per
upstream. Proven on this build: with two upstreams serving one slug and the
second-declared one 100× cheaper, "sort": "price" dispatched
to the cheaper upstream first.
Two limits worth knowing. The latency window lives in one process and starts empty, so
an upstream that has not yet served a streaming request has no measurement and falls
back to declaration order; only legs that reached their streaming commit point record a
sample. And "throughput" is not supported — deliberately, because ranking
endpoints on one instance's traffic cannot match a platform observing everyone's.
Legs that can never dispatch
Expansion is optimistic: it produces a leg for every mapping, then the loop discovers that some of them cannot be attempted. Two classes matter during an incident, because both look like a chain that is shorter than the config implies:
-
Dialect skips. A leg whose upstream cannot express what the request
carries is skipped before anything is built: an
input_audiopart on a translated dialect, an image form the target dialect cannot carry, a file attachment anywhere butopenai-compat. The pre-flight check costs nothing, never fetches a URL, and annotatesunsupported_contenton the leg. It surfaces astranslated_bad_request, marked not dispatched. The exact per-dialect matrix is on Providers. -
Surface skips.
/v1/embeddingsruns through this same engine but isopenai-compatonly: a leg to any other dialect is a silentno_routeskip, and there is no streaming. This is a stated limitation, not a bug.
Note
Put translated_bad_request in advanceOn whenever a chain
contains a non-compat upstream. Without it, a pre-flight content skip surfaces to the
client instead of falling through to the next leg.
Verdicts: retryable, fatal, and what the client sees
Every leg outcome is classified into exactly one verdict before any policy runs. Verdicts
are a closed set, and the names below are the literal strings you will read in the routing
report (retryable/<reason> or fatal/<reason>), in
advanceOn, in sameLegRetries, and on the
switchback_leg_attempts_total metric.
The reason taxonomy
| Reason | Retryable | What produced it |
|---|---|---|
| rate_limited | Yes |
A 429. Any Retry-After the profile can read is honored, capped.
|
| auth_budget | Yes | A 401/402/403-class provider permission or provider-side budget error. |
| server_error | Yes | A 5xx. |
| connect_error | Yes | TCP, TLS or DNS failure, including a transport failure while reading the body. |
| timeout_connect | Yes | The connect deadline elapsed. |
| timeout_ttft | Yes | Dispatched, then stalled before the first token arrived. |
| silent_empty | Yes | A 200 with no content, and a finish_reason of stop or absent. |
| stream_error_pre_token | Yes | A 200 stream whose very first event is an error frame. |
| translated_bad_request | Yes | A 400 on a translated leg, or a pre-flight unexpressible-content skip. |
| budget_exhausted | Only under overflowFallback |
The router's own reserve step refused the leg. Retryable only when
budget.overflowFallback is on; otherwise the fatal form below.
|
| bad_request | Fatal | A 4xx on a compat leg, and any non-400 4xx anywhere. |
| content_policy | Fatal | The upstream flagged moderation, per that host profile's moderation path. |
| budget_exhausted | Fatal |
The router's own cap refused the leg, with overflowFallback off (the
default).
|
| no_candidates | Fatal | The chain was exhausted with nothing to surface, because every leg was a silent skip. |
The split is not a severity judgement, it is a question of whether another upstream could plausibly do better. A malformed request is the caller's, so retrying it against three providers just burns three sets of credentials; a 429 is the provider's, so the next provider is exactly the right move.
Upstream status → verdict
| Upstream status | Verdict |
|---|---|
| 2xx |
Inspect the body; content decides between success and silent_empty or
stream_error_pre_token
|
| 429 |
retryable rate_limited, plus retry_after_ms read via the
profile's retry headers
|
| 401 / 402 / 403 | retryable auth_budget |
| 400 |
retryable translated_bad_request on a translated leg; otherwise
fatal bad_request
|
| other 4xx | fatal bad_request |
| 5xx | retryable server_error |
No provider's error grammar is hard-coded into the classifier. A host profile supplies the
envelope shape, the retry-header formats, and reasonHints that can remap a
provider's own code or type onto any reason in the taxonomy. The
openrouter profile, for instance, maps codes 402 and
403 onto auth_budget. A profile's moderation flag path is what
produces fatal content_policy.
Verdict → client HTTP status
When the chain gives up, the surfaced error prefers the upstream's own 4xx or 5xx status if there was one. Otherwise the verdict decides:
| Verdict | Status |
|---|---|
| rate_limited | 429 |
| auth_budget | 402 |
| bad_request, translated_bad_request, content_policy | 400 |
| server_error, silent_empty, stream_error_pre_token, no_candidates | 502 |
| connect_error, timeout_connect, timeout_ttft | 504 |
| budget_exhausted | 402 always |
budget_exhausted is the one verdict that ignores the upstream status
entirely: it is always 402, and always carries the budget envelope described
in Budgets.
The fallthrough decision
For a non-success outcome the policy is evaluated in this order, and the first rule that applies wins. Reading it top to bottom is how you explain any "why didn't it fail over?" question.
-
successcommits and the request is over. Ano_routeskip advances immediately, 0 ms. Anyfatalverdict surfaces. -
budget_exhaustedadvances ifbudget.overflowFallbackis on, and surfaces if it is not.advanceOnis not consulted for this reason at all. -
A reason not listed in
advanceOnsurfaces immediately, before the attempt budget is even considered. - If
maxTotalAttemptshas been reached, surface. -
If
sameLegRetries[reason]has retries left, retry the same leg after a wait. - Otherwise advance to the next leg, 0 ms.
Advancing to a different upstream never waits. Only a same-leg retry sleeps, which is the behavior you want at 3am: a failing provider costs you one attempt, not a backoff ladder, before the next one is tried.
Waiting: same-leg backoff and Retry-After
sameLegRetries is a per-reason count and
defaults to 0 for every reason. That default is the single most
surprising thing about the policy: without an explicit entry, a retryable verdict does not
retry its own upstream at all. It advances.
{"sameLegRetries": {"server_error": 1}} is what buys you one in-place retry
on a 5xx.
When a same-leg retry does happen, the wait is whichever applies:
-
The upstream's own
Retry-Afterhint, if the profile could read one and it is within the cap. -
Otherwise jittered exponential backoff: a ceiling of
min(maxMs, baseMs × 2^(n-1)), with the actual wait drawn uniformly below that ceiling. Defaults arebaseMs: 200,maxMs: 2000,jitter: true.
Warning
Retry-After is always capped. If the route sets no
respectRetryAfter.capMs, a built-in cap of 30000 ms applies. A
hint above the cap does not clamp to the cap. It causes an
immediate advance instead of a sleep. A provider asking you to wait ten
minutes will never hold a request open for ten minutes.
Attempt accounting
maxTotalAttempts counts dispatched attempts only. The
counter increments when a leg actually reached an upstream, so the three ways a leg can
end without being dispatched are free:
- a
no_routeskip (no mapping, or a surface the dialect cannot serve), - a budget reserve refusal, and
- a pre-flight expressibility skip.
None of those can consume the budget and truncate a reachable leg further down the chain.
Expansion deliberately does not pre-slice the leg list to
maxTotalAttempts either: a chain of eight legs where the first five are skips
still gets to try the sixth under maxTotalAttempts: 4. Set it to bound real
provider calls, and read it as "how many times am I willing to actually pay someone", not
"how many chain positions do I walk".
On the way out, x-switchback-attempts on a committed response is the
sum of per-leg attempts across the whole request. Three legs tried once
each is 3, one leg retried twice then a second leg is also 3. In
the report, a leg's own attempts key is emitted
only when it is greater than 1, so its absence means exactly one attempt.
To tell those two shapes apart, count the leg entries; see
Observability.
Note
Default is unlimited. With no maxTotalAttempts, the chain is bounded only
by its own length and by sameLegRetries.
Worked example: the first upstream rate-limits
A two-upstream failover route. One model slug is mapped explicitly to both upstreams; a
second candidate is served only by the fallback, through its
unlistedModels rule.
{
"route": {
"listen": { "port": 4011 },
"upstreams": {
"openai-direct": {
"baseURL": "https://api.openai.com/v1",
"dialect": "openai-compat",
"profile": "openai",
"auth": { "env": "OPENAI_API_KEY" }
},
"openrouter-fallback": {
"baseURL": "https://openrouter.ai/api/v1",
"dialect": "openai-compat",
"profile": "openrouter",
"auth": { "env": "OPENROUTER_API_KEY" },
"unlistedModels": "verbatim"
}
},
"models": {
"gpt-4o": {
"openai-direct": "gpt-4o",
"openrouter-fallback": "openai/gpt-4o"
}
},
"routes": {
"chat-default": {
"match": { "surface": ["/v1/chat/completions", "/api/v1/chat/completions"] },
"chain": ["gpt-4o", "openrouter/auto"],
"provider": { "allow_fallbacks": true },
"policy": {
"advanceOn": ["rate_limited", "server_error", "timeout_ttft", "silent_empty"],
"sameLegRetries": { "server_error": 1 },
"respectRetryAfter": { "capMs": 5000 },
"maxTotalAttempts": 4
}
}
}
}
}
For a request with no models[], that config expands to exactly three legs, in
this order:
| # | Candidate | Upstream | Upstream model id | Why |
|---|---|---|---|---|
| 1 | gpt-4o | openai-direct | gpt-4o | Explicit mapping, first upstream key |
| 2 | gpt-4o | openrouter-fallback | openai/gpt-4o | Explicit mapping, second upstream key |
| 3 | openrouter/auto | openrouter-fallback | openrouter/auto |
Unlisted slug, opted in by unlistedModels: "verbatim".
openai-direct has no rule, so it contributes nothing
|
Now the caller posts to /v1/chat/completions with no models[],
and OpenAI is rate-limiting.
-
Route selection. The path is in
match.surface, sochat-defaultadmits the request. -
Candidates. No
models[]in the body, so the chain is used verbatim:["gpt-4o", "openrouter/auto"]. The body'smodelvalue is recorded asrequested_modeland otherwise ignored. - Expansion. The three legs above.
-
Leg 1,
openai-direct. Budget reserves, the request dispatches, and OpenAI answers429withRetry-After: 12. Classified retryablerate_limited, withretry_after_ms: 12000. The reservation is released: a 429 never ran, so it never cost money. Dispatched attempts: 1. -
The decision. Not fatal, not a budget refusal.
rate_limitedis inadvanceOn, so it does not surface.maxTotalAttempts: 4is not reached.sameLegRetrieshas an entry forserver_erroronly, sorate_limitedhas zero same-leg retries: the engine advances, waiting 0 ms. The 12-second hint is never slept on; it would not have been anyway, since 12000 ms exceeds this route'scapMs: 5000. -
Leg 2,
openrouter-fallback. Same candidate, different upstream, model id rewritten toopenai/gpt-4o. It returns200with content, which classifies assuccessand commits. Dispatched attempts: 2. - Leg 3 is never touched. Commit stops the engine; there is no re-routing after a winner.
HTTP/1.1 200 OK
x-switchback-upstream: openrouter-fallback
x-switchback-dialect: openai-compat
x-switchback-attempts: 2
What the report shows
Two leg entries, in the order they were tried, each carrying its own verdict. Neither
entry has an attempts key, because each was attempted exactly once. The
2 in the header is the sum across legs.
{
"id": "rt-4b1f0a92c7de",
"surface": "/v1/chat/completions",
"requested_model": "gpt-4o",
"candidates": ["gpt-4o", "openrouter/auto"],
"legs": [
{
"candidate": "gpt-4o",
"upstream": "openai-direct",
"dialect": "openai-compat",
"verdict": "retryable/rate_limited",
"status": 429,
"retry_after_ms": 12000,
"budget": { "settlement": "released" }
},
{
"candidate": "gpt-4o",
"upstream": "openrouter-fallback",
"dialect": "openai-compat",
"verdict": "success",
"status": 200,
"budget": { "settlement": "committed_actuals" }
}
],
"client": { "aborted": false },
"winner": {
"candidate": "gpt-4o",
"upstream": "openrouter-fallback",
"provider": "openrouter"
},
"budget": { "scope": "global", "remaining": null }
}
winner.provider is openrouter because provider attribution comes
from the upstream's profile, falling back to the upstream's name.
budget.remaining is null because this config declares no dollar
cap.
Variations worth knowing
-
Leg 2 also 429s. Same decision, so the engine advances to leg 3. If
that 429s too, the chain is exhausted and the last upstream error surfaces:
here a
429, because the upstream's own status is preferred. -
Leg 2 returns 400. Both upstreams are
openai-compat, so a 400 is fatalbad_request. It surfaces immediately and leg 3 is never tried, no matter whatadvanceOnsays. Fatal beats the chain. -
Leg 1 returns 500 instead.
server_errorhassameLegRetries: 1, soopenai-directis retried in place first, after a jittered wait under the 2000 ms default ceiling. That costs a second dispatched attempt againstmaxTotalAttempts: 4. -
The caller sends
"models": ["openrouter/auto"]. Candidates become[model, "openrouter/auto"], the request's own list, not the route'schain. That is the only way a caller changes the candidate set. -
A dollar cap refuses leg 1 at reserve time. Nothing dispatched, nothing
counted against
maxTotalAttempts. WithoverflowFallback: false(the default), the client gets402immediately, even thoughadvanceOnis irrelevant to that decision.
Commit, and the end of the failover window
Exactly one leg wins, and the engine stops there. That makes the failover window precise and worth memorising:
Warning
Failover is pre-first-token only. Once a streaming response has
committed, the router never re-routes. A failure after that point is delivered in-stream
as an SSE event: error frame, so your client has to handle a stream that
starts fine and ends badly.
-
Client aborts before commit. The report's
client.abortedis set and the last real verdict surfaces. -
Chain exhausted with no winner. The last upstream error surfaces, or
fatal
no_candidatesif every leg was a silent skip and there is no upstream error to report.
Reading a route during an incident
Symptom to first place to look, in the order that usually pays off:
| Symptom | Where to look first |
|---|---|
Everything 404s with no_route_configured |
match.surface: exact paths, and both surface forms if clients use
both
|
404 no upstream mapping for candidates |
route.models for those slugs, and every upstream's
unlistedModels rule
|
| An error surfaces without trying the rest of the chain |
Was the verdict fatal? If retryable, is that reason in
advanceOn? Both surface at step 1 or 3 of the decision
|
| The report shows fewer legs than the config implies |
Silent skips. no_route legs leave no report entry at
all; expressibility skips leave one with unsupported_content
|
| The wrong upstream keeps serving traffic |
Declaration order plus provider.order. Not sort, which
does nothing
|
A 402 nobody expected |
budget_exhausted (the router's own cap) versus
auth_budget (the provider's). The envelope tells you which; see
Budgets
|
| Requests hang longer than a provider timeout |
Same-leg retries and their waits. Advancing never sleeps, so a long total is
sameLegRetries plus Retry-After under the cap
|
Known traps
-
provider.sortdoes nothing. Ordering comes from declaration order, withprovider.orderas the only override. -
budget_exhaustedinadvanceOnhas no effect. Usebudget.overflowFallbackinstead. -
match.surfaceis exact,match.modelsis glob. -
A
no_routeleg leaves no trace in the report. A chain that looks shorter than expected almost always means silent skips. -
sameLegRetriesdefaults to 0. Without it, a retryable verdict advances rather than retrying the same upstream. -
Without
models[], the request'smodeldoes not choose a candidate. The route'schaindoes. -
/v1/embeddingsisopenai-compatonly. Other dialects are silent skips, and there is no streaming.
The key-by-key reference for routes, provider and
policy is in Configuration; which request
features survive a translated leg (and therefore which legs are viable at all) is in
Providers; the verdict names show up again in the report and the
metrics on Observability. If you are coming from OpenRouter,
OpenRouter migration covers which
provider{} preferences carry over.