switchback — part of the aimock family private · pre-launch

Budgets

A budget is a dollar or token cap that the router checks before it contacts an upstream, and reconciles against real usage afterwards. When a cap binds, the caller gets a 402 and no provider is ever called, so the money you did not authorize is money you never spend.

What a cap actually promises

Every dispatched leg passes through a ledger operation called a reservation. The router estimates what the leg will cost, and admits it only if this holds, atomically, for every scope the leg touches:

the enforced invariant per scope, per reserve
spent + reserved + estimate <= cap

That invariant is what makes concurrent requests unable to each individually "fit" and collectively blow past a cap: an in-flight leg's estimate occupies the cap for as long as it is in flight, and is replaced by the real cost once the leg settles.

Warning

A cap is not a hard ceiling on realized spend. It is a hard gate on new dispatch. Overshoot is bounded by the sum, over in-flight legs, of max(0, actualCost - reservedEstimate). That is nonzero whenever true tokenization exceeds the estimator's heuristic (dense code and CJK text can approach one token per character), whenever completionReserveFraction is below 1.0, or whenever a configured price is simply wrong. Size your cap with that headroom in mind rather than treating it as a fuse.

Note

The whole budget block is optional. Omit it and the router does no accounting at all: no caps, no estimation, no 402, and GET /budget reports an empty scope list.

Scopes

Caps are declared at three scopes, each of which becomes a ledger key. All three are independent accounts; a leg reserves against all of them at once.

Config Scope key Bounds
budget.global global The whole process, all traffic.
budget.perRoute.<name> route:<name> Everything matched by one named route.
budget.perUpstream.<name> upstream:<name> Everything dispatched to one upstream, from any route.

The reserve step walks the scopes in a fixed order (global, then route:<routeName>, then upstream:<legUpstream>), and a refusal names the first binding scope in that order. Scopes with no configured cap are skipped entirely, so a deployment that only sets budget.global pays for exactly one account lookup per leg.

Reporting uses a deliberately narrower precedence. The x-switchback-budget-remaining header and the budget block of a routing report pick a single binding scope: route, then upstream, then global, taking the first of those that has a configured cap. So a request can be gated by global while its header reports headroom on the tighter route: account. Read GET /budget when you want all scopes at once.

Windows

Every cap has a window, and every boundary is UTC. There are five, not three, and the default when window is omitted is monthly.

Window Rolls at Notes
hourly Top of the UTC hour
daily UTC midnight
weekly UTC midnight, Monday-anchored Not "seven days ago"; a fixed calendar week.
monthly First of the UTC month The default.
total Never A non-rolling lifetime cap anchored at the epoch. Its resetsAt is reported as 9999-12-31T23:59:59.999Z.

What a boundary does

Nothing is prorated and nothing carries over. When a scope's window rolls, that scope's spend and its outstanding reservations are both reset to zero and the new window starts clean. A request refused at 23:59:59 UTC under a daily cap succeeds a second later, at the full cap, with no back-pressure and no queue. There is no soft landing as a window fills, either: legs are admitted at full speed until one no longer fits, and then that one is refused.

Because the boundary discards in-flight reservations, a leg that spans a boundary is committed against the new window when it settles. Windows shorter than your slowest request will therefore attribute a little spend to the window after the one that authorized it.

Warning

A total window genuinely never resets, including across restarts, provided the backend persists its state. On the default in-memory backend with no persistPath, a restart silently returns a lifetime cap to zero spent. If you mean "lifetime", pair total with a persistent backend.

Cap kinds: dollars, tokens, or both

A cap object is { dollars?, tokens?, window? }. You may set either dimension, both, or neither. Both dimensions are enforced independently, so whichever runs out first is the one that refuses the request.

dollars string optional default: uncapped

A fixed-point decimal string, never a JSON number. Money is never a float anywhere in the router: dollars are parsed at the config boundary and carried internally as micro-dollar (1e-6) integers. The grammar is digits with an optional decimal point and at most six fractional digits. A leading -, a non-numeric string, "Infinity", "NaN", scientific notation such as "1e-6", or a seventh fractional digit are each rejected at load, so a malformed cap fails startup instead of silently becoming zero or infinity.

tokens integer optional default: uncapped

A non-negative integer counting prompt plus completion tokens together. A token cap needs no pricing at all, which makes it the enforcement you can always have, including for models the price catalog has never heard of.

window enum optional default: "monthly"

One of hourly, daily, weekly, monthly, total. Anything else is refused at load.

Reserve, then commit or release

The ledger exposes four operations (reserve, commit, release, read), and the router uses them in that shape around every leg.

Reserve is two-phase, and side-effect-free on refusal

Phase one evaluates the invariant for every applicable scope against a read-only view of the ledger. If any scope fails, nothing is mutated (no partial reservation is left behind on the scopes that would have fit) and a structured refusal is returned. Phase two runs only after every scope passes, materializing the accounts and recording the reservation under each. There is no window in which a refused request has reserved half a cap.

The refusal carries exactly these fields, and they are what the 402 body is built from:

Field Meaning
scope First binding scope key.
window That scope's window.
limitKind Which dimension bound: "dollars" or "tokens".
limit The configured cap, echoed verbatim.
spent Spend plus outstanding reservations at the moment of refusal.
resetsAt ISO-8601 UTC end of the window.

Reservations expire

A reservation has a TTL of 60 000 ms. Expired reservations are pruned on every ledger operation, and the in-memory backend additionally sweeps on a timer (60 s by default), so a crashed or abandoned request cannot hold a cap hostage forever. If a commit arrives for a reservation that was already pruned, it degrades gracefully: the real cost is still booked, as direct spend rather than as the settlement of a reservation. Nothing is lost from the ledger, but a request slower than the TTL does briefly stop occupying the cap, which is one of the ways two very slow concurrent legs can overshoot.

Which failures still cost money

After the leg finishes, its verdict selects a settlement. The rule is deliberately conservative: a leg the provider plausibly executed is charged, because a retry storm must not multiply unaccounted spend. Only failures that provably never reached a model release their reservation.

Leg verdict Settlement Charged?
success commit actuals (reported usage, priced) Yes
success commit relayed_tokens when the stream ended abnormally after commit and carried no usage frame. Never a plain release after commit. Yes
rate_limited release No
auth_budget release No
connect_error release No
timeout_connect release No
server_error commit: error_usage if the error body carried usage, else prompt_estimate Yes
silent_empty commit, same rule Yes
timeout_ttft commit, same rule Yes
stream_error_pre_token commit, same rule Yes
translated_bad_request release if the leg was dispatched, else none Only if dispatched
bad_request release No
content_policy commit, same rule as server_error Yes
budget_exhausted none: the refusal happened at reserve time, so there is nothing to settle No
no_candidates none No

The settlement chosen for each leg is visible per-leg in the routing report as budget: { settlement, amount }, using the labels released, committed_actuals, committed_prompt_estimate, committed_error_usage, committed_relayed_tokens and none. See Observability for the full report shape, and Routing for where those verdicts come from.

Note

At settle time the router prefers the upstream's own reported cost when it is a non-negative finite number; failing that it prices the reported token counts from the catalog or your overlay; failing that it keeps the estimate. The chosen source is stamped on the report as cost: { amount, source }, so you can tell a provider-reported figure from a locally priced one.

How a leg is estimated

A reservation needs a number before a single token exists, so the router estimates one. Two different precision regimes meet here on purpose: token counts are a heuristic, widened by a safety factor; money is exact, computed in fixed-point integers and narrowed to micro-dollars exactly once.

estimateCost pseudocode
estimate = promptEstimate    × promptPrice
         + imageCount        × imagePrice
         + completionCeiling × completionPrice

promptEstimate    = ceil(ceil(serializedChars / 4) × promptSafetyFactor)
completionCeiling = ceil((max_tokens ?? 4096) × completionReserveFraction)
Constant Default
promptSafetyFactor 2.0
completionReserveFraction 1.0
catalog max completion fallback 4096
per-image price "0.001275"

serializedChars counts everything the upstream will tokenize, not just user text: all message content including the system message, each message's name and tool_call_id, the serialized tool_calls history, and the serialized tools schemas. Image parts are excluded from the character count and priced per image instead. Caller-supplied numbers are clamped before they reach the money math: a missing or non-finite max_tokens falls back to the catalog default, a finite negative floors to zero, and token counts coerce to non-negative integers.

Warning

promptSafetyFactor and completionReserveFraction are the two knobs that decide how much a cap can be overshot. Lowering either buys you throughput near a full cap and gives back precision: at completionReserveFraction: 0.25 the router reserves a quarter of the requested max_tokens, so four maximally verbose concurrent completions can settle for more than was ever reserved. Raise promptSafetyFactor instead if your traffic is dense code or non-Latin script, where chars / 4 under-counts badly.

For /v1/embeddings, completionReserveFraction is forced to 0. There is no completion to reserve, so estimation is prompt-side only.

Pricing: the catalog and the priceMap overlay

A dollar cap can only bind if the leg has a price. Prices come from two places: a signed catalog artifact bundled with the router, and your own priceMap overlay, which always wins.

The catalog is verified as a detached Ed25519 signature before it is parsed, at every load, so a tampered catalog never partial-loads. Pricing is a money surface and is treated as one. Catalog rows are keyed canonically, like openai/gpt-4o, with aliases such as gpt-4o; both resolve, and canonical keys win over aliases.

Danger

priceMap keys are <upstreamModelId>@<upstream>: the id the upstream knows, not the slug your clients send. A key that does not match produces no error and no warning: the overlay is silently ignored, the leg falls through to the catalog, and your price override quietly never applies. If a dollar cap is configured, that near-miss key is the difference between a free local model and a leg refused as unpriceable. Where the catalog does have a row, it is the difference between your negotiated rate and a list price. Check every key you add against the right-hand side of your models mapping.

the mapping decides the key json
{
  "models": {
    // clients ask for this slug …
    "llama-3.3-70b": {
      // … and this upstream calls it this.
      "ollama-local": "llama3.3:70b"
    }
  },

  "budget": {
    "priceMap": {
      // RIGHT: upstreamModelId@upstream
      "llama3.3:70b@ollama-local": { "prompt": "0", "completion": "0" },

      // WRONG: the client slug. Never matches. Never warns.
      "llama-3.3-70b@ollama-local": { "prompt": "0", "completion": "0" }
    }
  }
}

An overlay entry accepts prompt, completion, and optionally cache_read and image. All of them are plain non-negative decimal strings under the same grammar as a dollar cap; scientific notation is rejected at load with an explicit message. A malformed price literal is never treated as $0. It is either refused at load or leaves the model unpriced.

The router never guesses a price

If a leg has no price, and a dollar cap applies to it, and allowUnpriced is false (the default), the leg is refused with budget_unpriceable, and the error names the two remedies: set allowUnpriced: true, or add a priceMap entry. With no dollar cap, or with allowUnpriced: true, an unpriced leg proceeds normally and any token cap still applies to it.

Danger

The bundled catalog prices only the models it lists. A daily job rebuilds it from OpenRouter and vendor price pages and opens a reviewed pull request, so your release prices the models in its last merged refresh. A model listed at $0 prompt and $0 completion counts as unpriced unless its id ends in :free. With a dollar cap configured and allowUnpriced left at false, every unpriced model refuses at reserve time with budget_unpriceable until you give it a priceMap entry. This is the single biggest practical limit of budgets today. Plan for it: either price every model you route to, or lean on tokens caps, which need no prices at all. Setting allowUnpriced: true to make the errors go away means unpriced legs draw down nothing from your dollar cap. The cap stops binding for exactly the traffic you could not price.

A budget block you can paste

This is the whole feature surface in one object, nested under the same root as the rest of your configuration.

switchback.config.json json
{
  "budget": {
    "backend": { "kind": "memory", "persistPath": "./.switchback/budget.json" },

    "global": { "dollars": "500.00", "tokens": 50000000, "window": "monthly" },
    "perRoute": { "chat-default": { "dollars": "50.00", "window": "daily" } },
    "perUpstream": { "openrouter-fallback": { "dollars": "25.00", "window": "monthly" } },

    "priceMap": {
      "llama3.3:70b@ollama-local": { "prompt": "0", "completion": "0" }
    },

    "promptSafetyFactor": 2.0,
    "completionReserveFraction": 1.0,
    "allowUnpriced": false,
    "overflowFallback": false,
    "onStoreError": "fail-closed"
  }
}

Note

On SIGHUP the config file is re-read and budget caps, the priceMap overlay and the cap gauges are rebuilt in place. The budget backend is not: changing backend.kind, its url or its persistPath logs a loud "restart required" warning and keeps running the old store. Cap changes are hot; topology changes are not.

Backends

The ledger is pluggable because the correct answer depends entirely on how many processes share a cap. Pick by topology, not by preference.

backend.kind Use when What you are trading
memory default One process owns the cap. Atomicity is free: the critical section is single-threaded with no await in it. Optional persistPath snapshot, written every 30 000 ms and again on shutdown, so a crash can lose up to that window of spend.
file Several processes on one host share a cap. Correct but low-throughput: an advisory lockfile plus read-modify-write plus a temp-file rename on every operation. Lock timeout 5 000 ms, stale-lock steal after 10 000 ms. persistPath is required; the CLI refuses to start without it.
redis A fleet of replicas shares one cap. The only fleet-shared component in the router. reserve, commit and release are each a single Lua script, money crosses the wire as micro-dollar integers, and reservation expiry is enforced inside the script rather than by key TTL.

A per-replica memory backend in a fleet does not enforce your cap; it enforces N copies of your cap. That is the one substitution that silently multiplies spend, and it is why Deploy treats the budget backend as a topology decision.

State files are refused if they are readable by anyone else

A budget snapshot records your spend, so both the memory and file backends harden it. A state file that is group- or other-accessible (mode & 0o077) is refused at load, with the offending octal mode in the message. A corrupt or wrong-shaped snapshot also refuses to boot rather than silently starting from a zeroed ledger. A budget store that quietly forgets its spend is worse than one that will not start. Snapshots are written 0600 with an explicit chmod.

Redis specifics

Keys are prefixed {switchback-budget}. The braces are a Cluster hash tag, so every scope key lands in one slot. Cluster MOVED and ASK redirects are not followed, so supported topologies are standalone, Sentinel behind a proxy, or Cluster behind a single-endpoint proxy. The connection string goes in backend.url as a redis:// or rediss:// URL; keep a real credential out of any config file you commit.

Danger

onStoreError: "fail-open" means that when the store is unreachable, caps are not enforced and requests keep flowing. It is loud about it: the log line reads [switchback][budget] fail-open: cap NOT enforced. But /ready still answers ready, with "budgetStore":"degraded". The default, "fail-closed", instead refuses traffic and takes /ready to 503 so your orchestrator stops sending it work. Choose deliberately: fail-open trades a cap for availability, and a Redis outage during a runaway workload is exactly when the cap mattered.

At the cap: the 402

Refusal happens at reserve time, which is before any upstream is contacted. The status is always 402 Payment Required for budget_exhausted, on both the native and the OpenRouter-shaped surface.

POST /v1/chat/completions under a spent cap http
HTTP/1.1 402 Payment Required
content-type: application/json
x-switchback-budget-remaining: global=0.000001

{
  "error": {
    "message": "budget cap reached for scope \"global\" (monthly); limit 0.000001, spent 0.000000, resets 2026-08-01T00:00:00.000Z",
    "type": "budget_exhausted",
    "code": 402,
    "metadata": {
      "reason": "budget_exhausted",
      "scope": "global",
      "window": "monthly",
      "limit_kind": "dollars",
      "limit": "0.000001",
      "spent": "0.000000",
      "remaining": "0.000001",
      "resets_at": "2026-08-01T00:00:00.000Z"
    }
  }
}

The same request against the /api/v1 surface returns the OpenRouter-shaped error envelope carrying the identical metadata object, so a client written against either shape can branch on metadata.reason and back off until metadata.resets_at. The metadata keys are exactly reason, scope, window, limit_kind, limit, spent, remaining, resets_at.

Note that spent is spend plus outstanding reservations, which is why a 402 can report a spent lower than the limit: the difference is in-flight work that has already claimed the remaining headroom.

Warning

An in-flight stream is never cut off when a cap is crossed mid-stream. The enforcement point is reserve, before dispatch, so caps gate new dispatch only. A completion that was admitted runs to its end and is charged in full. Likewise, listing budget_exhausted in a route's advanceOn does nothing: it is ignored there, and governed by overflowFallback instead.

overflowFallback

By default a budget refusal is terminal: the 402 surfaces immediately and the chain stops. That is usually what you want, because the alternative spends money you were trying not to spend.

Set overflowFallback: true and the engine instead advances to the next leg, whose own estimate is checked against whatever headroom remains. A leg reached this way is annotated budget_overflow_fallback: true in the report and the response carries x-switchback-budget-overflow: true. If nothing in the chain fits, the last refusal surfaces as the 402. This is how you express "when the expensive upstream's per-upstream cap is spent, fall through to the cheap one": the caps do the routing. It only helps if the next leg is cheaper or sits under a different scope; if the binding scope is global, no leg can fit, and the overflow just walks the chain to the same refusal.

The remaining-budget header

Successful responses can carry x-switchback-budget-remaining, formatted <scope>=<amount> in six-decimal fixed point, for example x-switchback-budget-remaining: global=4.999977. It is a convenience for clients that want to self-throttle, and it is honest about its own limits:

Reading current spend

GET /budget is the authoritative view: read straight from the store, credential-scrubbed on the way out, listing only the scopes your config declares, ordered global then upstream:* then route:*.

current spend against caps shell
$ curl -s http://127.0.0.1:4011/budget
response json
{
  "budgetBackend": "memory",
  "scopes": [
    {
      "scope": "global",
      "window": "monthly",
      "windowStart": "2026-07-01T00:00:00.000Z",
      "spentTokens": 0,
      "spentDollars": "0.000000",
      "reservedTokens": 0,
      "reservedDollars": "0.000000",
      "limit": { "dollars": "5.00" },
      "remaining": { "dollars": "5.000000" }
    }
  ]
}

remaining.dollars is limit − (spent + reserved), so it is what a new request would actually be measured against rather than a pure spend figure. A store read failure answers 503 budget store is unavailable, never a raw 500. An uncapped deployment answers with an empty scopes array.

For scraping, /metrics exposes switchback_budget_cap and switchback_budget_spent gauges, each labelled with scope and nothing else. They emit only when a cap lookup is available and the binding scope's remaining is non-null, and they do correctly report a scope that has gone over its cap. The per-request settlement trail lives in Observability.

Known limits

Everything here is current behavior, not a roadmap. Read it before you rely on a cap for anything that matters.

  1. Two priced models out of the box. Under a dollar cap with allowUnpriced: false, anything outside the bundled catalog needs a priceMap entry or it is refused.
  2. A wrong priceMap key fails silently. The key is <upstreamModelId>@<upstream>. There is no warning for a near-miss.
  3. Caps bound new dispatch, not realized spend. Overshoot is bounded, not zero.
  4. A fail-open store means no enforcement. Loud in logs, degraded on /ready, but not enforcing.
  5. Two deferred defects worth knowing. A malformed numeric field read back from Redis throws past onStoreError (producing a 500 even in fail-open mode), and the file-store's lock rename-back has a narrow re-clobber race. Both are tracked as known hardening work.
  6. /budget only shows declared scopes. An empty list means no caps are configured, not that nothing has been spent.

Next: the budget block field-by-field in Configuration, which backend suits your process topology in Deploy, and where settlements, gauges and reports surface in Observability.