Routing between models

Models differ a lot in price, speed and ability. Sending everything to the biggest one is simple and expensive; sending everything to the smallest is cheap and sometimes wrong. Routing sends each request to the model that fits it.

Simple rules go a long way: short classification and extraction tasks to a small, fast model; long documents, hard reasoning and anything customer-facing that failed a check to a larger one. A cheap classifier (a small model, or ordinary code) can make the call when rules can't.

A cascade tries the cheap model first and escalates only when its output fails a check: invalid JSON, totals that don't add up, an abstention. If most requests pass at the first level, you pay big-model prices only for the hard ones.

requests = 100_000
small_cost, large_cost = 0.0004, 0.006   # dollars per request (made-up figures)
escalated = 0.15                          # share that fails the small model's checks

all_large = requests * large_cost
cascade = requests * small_cost + requests * escalated * large_cost
print(round(all_large), round(cascade))

Have a fallback for outages: if the main provider or model is down or slow, route to a second one that has passed the same evals. Keep prompts portable enough that switching is a configuration change, and test the fallback regularly rather than discovering on the bad day that it doesn't work.

Measure the router itself. How often does it send a hard request to the small model (and get a bad answer) or an easy one to the big model (and waste money)? Run your eval set through the whole routed system, not just each model on its own, and re-check when any model is updated.