switchback — part of the aimock family private · pre-launch

Routing

One inbound request becomes an ordered list of attempts. This page is the mechanism, end to end: which route matches, which candidates it produces, how each candidate expands into legs, which failures retry the same upstream, which move on, which stop everything, and exactly what the attempt budget counts.

Note

Everything below is the behavior of the current build, which is private and pre-launch: the router prints an experimental notice on every start, and the config shape can still change. Where a key is accepted but does not yet do anything, this page says so.

The pipeline, in order

Five stages, always in this order. If you can name the stage a request died in, you have already narrowed the cause to one config block.

  1. Route selection. The first route in declaration order whose match admits the request wins. No match is a 404.
  2. Candidate chain. [model, ...models[]] when the request body carries a non-empty models[]; otherwise the route's chain, verbatim.
  3. Leg expansion. Each candidate becomes one leg per upstream that can serve it, filtered and ordered by provider{}.
  4. The loop. Per leg: reserve budget, dispatch, classify the outcome, settle the reservation. Then the policy says commit, retry the same leg, advance, or surface the error.
  5. Commit. The first successful leg wins and the engine stops. Nothing re-routes after that.

Route selection

Routes are declared as an object keyed by route name, and matching walks them in declaration order. The first admitting route wins, so put your specific routes above your catch-all. Two keys can appear in match, and they do not work the same way.

match.surface string[] optional default: matches every path

An exact list of request paths, tested by membership: not a glob, not a prefix. "/v1/chat/completions" does not admit /api/v1/chat/completions; list both if you serve both surfaces.

match.models string[] optional default: matches every model

Globs. Each entry is anchored and * becomes .*, so "gpt-*" admits gpt-4o. Tested against the request's model field.

An omitted key matches everything, so "match": {} is a catch-all. When no route admits the request the client gets HTTP 404 with type no_route_configured and the message no route configured for path "<path>" and model "<model>".

Warning

surface is exact-match and models is glob. It is the easiest pair in the config to get backwards, and getting it backwards looks like a routing outage: every request 404s with no_route_configured.

The candidate chain

A candidate is a model slug the router is willing to try for this request. There are exactly two sources, and only one of them is the request body.

how candidates are chosen logic
// request body carries a non-empty models[] …
candidates = [req.model, ...req.models]

// … otherwise the route's own chain is used, verbatim
candidates = route.chain

Warning

Without models[] in the request body, the request's model field does not select a candidate at all. It feeds route matching and the report's requested_model, and nothing else. A route with "chain": ["gpt-4o"] and "match": {} answers a request for {"model":"nope"} with 200 off the gpt-4o leg, verified on this build. If you want the caller to steer the chain, the caller has to send models[].

The candidate list is de-duplicated preserving first-seen order, so a slug repeated between model and models[] is tried once, in its earliest position.

Leg expansion

A leg is one concrete attempt: a candidate slug paired with one upstream and the model id that upstream knows it by. Expansion walks the candidates in chain order, and for each one the rule depends on whether the slug is declared in route.models.

A candidate that maps nowhere contributes zero legs and is skipped silently, with no error and no leg entry in the report. If the whole chain maps nowhere, the client gets HTTP 404 with no upstream mapping for candidates: <list>.

provider{} composition: a request narrows, never widens

Both the route and the request body may carry a provider{} block. They are combined per key, and the combination is deliberately one-directional: a caller can restrict the upstreams their request touches, but cannot reach an upstream the route did not already allow.

Key Config × request
order request order replaces config order when present
only intersection of both lists
ignore union of both lists
allow_fallbacks any explicit false wins; default true
sort request value, else config value, else none (and see the warning below)

allow_fallbacks: false truncates the chain to the primary candidate only, but all of that candidate's legs are still tried. It bounds which models you may be served, not how many upstreams are attempted for the model you asked for.

Request-side provider{} is coerced defensively: only string arrays survive, and sort must be exactly "price" or "latency". Anything else is discarded rather than honored.

Leg order comes from declaration order

Within a candidate, legs are ordered by three passes, in this fixed sequence:

  1. provider.order front-loads the upstreams it names, in the order given; the rest stay behind them.
  2. only and ignore filter what survived.
  3. sort runs last, and is inert (see below).

So the baseline order is the order you wrote things down: for a mapped slug, the order of upstream keys inside that slug's entry in route.models; for an unlisted slug, the order upstreams are declared in route.upstreams. The only lever that reorders legs today is provider.order.

Note

provider.sort orders a candidate's legs. "price" uses the catalog price — cheapest first, with an unpriced leg sorting last rather than guessed at. "latency" uses the rolling mean TTFT this process has observed per upstream. Proven on this build: with two upstreams serving one slug and the second-declared one 100× cheaper, "sort": "price" dispatched to the cheaper upstream first.

Two limits worth knowing. The latency window lives in one process and starts empty, so an upstream that has not yet served a streaming request has no measurement and falls back to declaration order; only legs that reached their streaming commit point record a sample. And "throughput" is not supported — deliberately, because ranking endpoints on one instance's traffic cannot match a platform observing everyone's.

Legs that can never dispatch

Expansion is optimistic: it produces a leg for every mapping, then the loop discovers that some of them cannot be attempted. Two classes matter during an incident, because both look like a chain that is shorter than the config implies:

Note

Put translated_bad_request in advanceOn whenever a chain contains a non-compat upstream. Without it, a pre-flight content skip surfaces to the client instead of falling through to the next leg.

Verdicts: retryable, fatal, and what the client sees

Every leg outcome is classified into exactly one verdict before any policy runs. Verdicts are a closed set, and the names below are the literal strings you will read in the routing report (retryable/<reason> or fatal/<reason>), in advanceOn, in sameLegRetries, and on the switchback_leg_attempts_total metric.

The reason taxonomy

Reason Retryable What produced it
rate_limited Yes A 429. Any Retry-After the profile can read is honored, capped.
auth_budget Yes A 401/402/403-class provider permission or provider-side budget error.
server_error Yes A 5xx.
connect_error Yes TCP, TLS or DNS failure, including a transport failure while reading the body.
timeout_connect Yes The connect deadline elapsed.
timeout_ttft Yes Dispatched, then stalled before the first token arrived.
silent_empty Yes A 200 with no content, and a finish_reason of stop or absent.
stream_error_pre_token Yes A 200 stream whose very first event is an error frame.
translated_bad_request Yes A 400 on a translated leg, or a pre-flight unexpressible-content skip.
budget_exhausted Only under overflowFallback The router's own reserve step refused the leg. Retryable only when budget.overflowFallback is on; otherwise the fatal form below.
bad_request Fatal A 4xx on a compat leg, and any non-400 4xx anywhere.
content_policy Fatal The upstream flagged moderation, per that host profile's moderation path.
budget_exhausted Fatal The router's own cap refused the leg, with overflowFallback off (the default).
no_candidates Fatal The chain was exhausted with nothing to surface, because every leg was a silent skip.

The split is not a severity judgement, it is a question of whether another upstream could plausibly do better. A malformed request is the caller's, so retrying it against three providers just burns three sets of credentials; a 429 is the provider's, so the next provider is exactly the right move.

Upstream status → verdict

Upstream status Verdict
2xx Inspect the body; content decides between success and silent_empty or stream_error_pre_token
429 retryable rate_limited, plus retry_after_ms read via the profile's retry headers
401 / 402 / 403 retryable auth_budget
400 retryable translated_bad_request on a translated leg; otherwise fatal bad_request
other 4xx fatal bad_request
5xx retryable server_error

No provider's error grammar is hard-coded into the classifier. A host profile supplies the envelope shape, the retry-header formats, and reasonHints that can remap a provider's own code or type onto any reason in the taxonomy. The openrouter profile, for instance, maps codes 402 and 403 onto auth_budget. A profile's moderation flag path is what produces fatal content_policy.

Verdict → client HTTP status

When the chain gives up, the surfaced error prefers the upstream's own 4xx or 5xx status if there was one. Otherwise the verdict decides:

Verdict Status
rate_limited 429
auth_budget 402
bad_request, translated_bad_request, content_policy 400
server_error, silent_empty, stream_error_pre_token, no_candidates 502
connect_error, timeout_connect, timeout_ttft 504
budget_exhausted 402 always

budget_exhausted is the one verdict that ignores the upstream status entirely: it is always 402, and always carries the budget envelope described in Budgets.

The fallthrough decision

For a non-success outcome the policy is evaluated in this order, and the first rule that applies wins. Reading it top to bottom is how you explain any "why didn't it fail over?" question.

  1. success commits and the request is over. A no_route skip advances immediately, 0 ms. Any fatal verdict surfaces.
  2. budget_exhausted advances if budget.overflowFallback is on, and surfaces if it is not. advanceOn is not consulted for this reason at all.
  3. A reason not listed in advanceOn surfaces immediately, before the attempt budget is even considered.
  4. If maxTotalAttempts has been reached, surface.
  5. If sameLegRetries[reason] has retries left, retry the same leg after a wait.
  6. Otherwise advance to the next leg, 0 ms.

Advancing to a different upstream never waits. Only a same-leg retry sleeps, which is the behavior you want at 3am: a failing provider costs you one attempt, not a backoff ladder, before the next one is tried.

Waiting: same-leg backoff and Retry-After

sameLegRetries is a per-reason count and defaults to 0 for every reason. That default is the single most surprising thing about the policy: without an explicit entry, a retryable verdict does not retry its own upstream at all. It advances. {"sameLegRetries": {"server_error": 1}} is what buys you one in-place retry on a 5xx.

When a same-leg retry does happen, the wait is whichever applies:

Warning

Retry-After is always capped. If the route sets no respectRetryAfter.capMs, a built-in cap of 30000 ms applies. A hint above the cap does not clamp to the cap. It causes an immediate advance instead of a sleep. A provider asking you to wait ten minutes will never hold a request open for ten minutes.

Attempt accounting

maxTotalAttempts counts dispatched attempts only. The counter increments when a leg actually reached an upstream, so the three ways a leg can end without being dispatched are free:

None of those can consume the budget and truncate a reachable leg further down the chain. Expansion deliberately does not pre-slice the leg list to maxTotalAttempts either: a chain of eight legs where the first five are skips still gets to try the sixth under maxTotalAttempts: 4. Set it to bound real provider calls, and read it as "how many times am I willing to actually pay someone", not "how many chain positions do I walk".

On the way out, x-switchback-attempts on a committed response is the sum of per-leg attempts across the whole request. Three legs tried once each is 3, one leg retried twice then a second leg is also 3. In the report, a leg's own attempts key is emitted only when it is greater than 1, so its absence means exactly one attempt. To tell those two shapes apart, count the leg entries; see Observability.

Note

Default is unlimited. With no maxTotalAttempts, the chain is bounded only by its own length and by sameLegRetries.

Worked example: the first upstream rate-limits

A two-upstream failover route. One model slug is mapped explicitly to both upstreams; a second candidate is served only by the fallback, through its unlistedModels rule.

switchback.config.json json
{
  "route": {
    "listen": { "port": 4011 },
    "upstreams": {
      "openai-direct": {
        "baseURL": "https://api.openai.com/v1",
        "dialect": "openai-compat",
        "profile": "openai",
        "auth": { "env": "OPENAI_API_KEY" }
      },
      "openrouter-fallback": {
        "baseURL": "https://openrouter.ai/api/v1",
        "dialect": "openai-compat",
        "profile": "openrouter",
        "auth": { "env": "OPENROUTER_API_KEY" },
        "unlistedModels": "verbatim"
      }
    },
    "models": {
      "gpt-4o": {
        "openai-direct": "gpt-4o",
        "openrouter-fallback": "openai/gpt-4o"
      }
    },
    "routes": {
      "chat-default": {
        "match": { "surface": ["/v1/chat/completions", "/api/v1/chat/completions"] },
        "chain": ["gpt-4o", "openrouter/auto"],
        "provider": { "allow_fallbacks": true },
        "policy": {
          "advanceOn": ["rate_limited", "server_error", "timeout_ttft", "silent_empty"],
          "sameLegRetries": { "server_error": 1 },
          "respectRetryAfter": { "capMs": 5000 },
          "maxTotalAttempts": 4
        }
      }
    }
  }
}

For a request with no models[], that config expands to exactly three legs, in this order:

# Candidate Upstream Upstream model id Why
1 gpt-4o openai-direct gpt-4o Explicit mapping, first upstream key
2 gpt-4o openrouter-fallback openai/gpt-4o Explicit mapping, second upstream key
3 openrouter/auto openrouter-fallback openrouter/auto Unlisted slug, opted in by unlistedModels: "verbatim". openai-direct has no rule, so it contributes nothing

Now the caller posts to /v1/chat/completions with no models[], and OpenAI is rate-limiting.

  1. Route selection. The path is in match.surface, so chat-default admits the request.
  2. Candidates. No models[] in the body, so the chain is used verbatim: ["gpt-4o", "openrouter/auto"]. The body's model value is recorded as requested_model and otherwise ignored.
  3. Expansion. The three legs above.
  4. Leg 1, openai-direct. Budget reserves, the request dispatches, and OpenAI answers 429 with Retry-After: 12. Classified retryable rate_limited, with retry_after_ms: 12000. The reservation is released: a 429 never ran, so it never cost money. Dispatched attempts: 1.
  5. The decision. Not fatal, not a budget refusal. rate_limited is in advanceOn, so it does not surface. maxTotalAttempts: 4 is not reached. sameLegRetries has an entry for server_error only, so rate_limited has zero same-leg retries: the engine advances, waiting 0 ms. The 12-second hint is never slept on; it would not have been anyway, since 12000 ms exceeds this route's capMs: 5000.
  6. Leg 2, openrouter-fallback. Same candidate, different upstream, model id rewritten to openai/gpt-4o. It returns 200 with content, which classifies as success and commits. Dispatched attempts: 2.
  7. Leg 3 is never touched. Commit stops the engine; there is no re-routing after a winner.
what the client sees http
HTTP/1.1 200 OK
x-switchback-upstream: openrouter-fallback
x-switchback-dialect: openai-compat
x-switchback-attempts: 2

What the report shows

Two leg entries, in the order they were tried, each carrying its own verdict. Neither entry has an attempts key, because each was attempted exactly once. The 2 in the header is the sum across legs.

GET /route/reports (illustrative, no budget block configured) json
{
  "id": "rt-4b1f0a92c7de",
  "surface": "/v1/chat/completions",
  "requested_model": "gpt-4o",
  "candidates": ["gpt-4o", "openrouter/auto"],
  "legs": [
    {
      "candidate": "gpt-4o",
      "upstream": "openai-direct",
      "dialect": "openai-compat",
      "verdict": "retryable/rate_limited",
      "status": 429,
      "retry_after_ms": 12000,
      "budget": { "settlement": "released" }
    },
    {
      "candidate": "gpt-4o",
      "upstream": "openrouter-fallback",
      "dialect": "openai-compat",
      "verdict": "success",
      "status": 200,
      "budget": { "settlement": "committed_actuals" }
    }
  ],
  "client": { "aborted": false },
  "winner": {
    "candidate": "gpt-4o",
    "upstream": "openrouter-fallback",
    "provider": "openrouter"
  },
  "budget": { "scope": "global", "remaining": null }
}

winner.provider is openrouter because provider attribution comes from the upstream's profile, falling back to the upstream's name. budget.remaining is null because this config declares no dollar cap.

Variations worth knowing

Commit, and the end of the failover window

Exactly one leg wins, and the engine stops there. That makes the failover window precise and worth memorising:

Warning

Failover is pre-first-token only. Once a streaming response has committed, the router never re-routes. A failure after that point is delivered in-stream as an SSE event: error frame, so your client has to handle a stream that starts fine and ends badly.

Reading a route during an incident

Symptom to first place to look, in the order that usually pays off:

Symptom Where to look first
Everything 404s with no_route_configured match.surface: exact paths, and both surface forms if clients use both
404 no upstream mapping for candidates route.models for those slugs, and every upstream's unlistedModels rule
An error surfaces without trying the rest of the chain Was the verdict fatal? If retryable, is that reason in advanceOn? Both surface at step 1 or 3 of the decision
The report shows fewer legs than the config implies Silent skips. no_route legs leave no report entry at all; expressibility skips leave one with unsupported_content
The wrong upstream keeps serving traffic Declaration order plus provider.order. Not sort, which does nothing
A 402 nobody expected budget_exhausted (the router's own cap) versus auth_budget (the provider's). The envelope tells you which; see Budgets
Requests hang longer than a provider timeout Same-leg retries and their waits. Advancing never sleeps, so a long total is sameLegRetries plus Retry-After under the cap

Known traps

  1. provider.sort does nothing. Ordering comes from declaration order, with provider.order as the only override.
  2. budget_exhausted in advanceOn has no effect. Use budget.overflowFallback instead.
  3. match.surface is exact, match.models is glob.
  4. A no_route leg leaves no trace in the report. A chain that looks shorter than expected almost always means silent skips.
  5. sameLegRetries defaults to 0. Without it, a retryable verdict advances rather than retrying the same upstream.
  6. Without models[], the request's model does not choose a candidate. The route's chain does.
  7. /v1/embeddings is openai-compat only. Other dialects are silent skips, and there is no streaming.

The key-by-key reference for routes, provider and policy is in Configuration; which request features survive a translated leg (and therefore which legs are viable at all) is in Providers; the verdict names show up again in the report and the metrics on Observability. If you are coming from OpenRouter, OpenRouter migration covers which provider{} preferences carry over.