A smaller model can lower the cost of a suitable task, but the price of one call is not the price of the workflow. Routing needs to account for whether the cheaper path succeeds, how failures are detected and what happens when a stronger model must take over.
Split tasks by what can be checked
Start with bounded operations such as classifying a known input into a fixed set of categories or extracting a field from a well-defined document. Build an evaluator that can detect important mistakes. A broad task with no reliable success check is harder to route safely.
Keep a sample of difficult cases: unusual formatting, missing data, ambiguous instructions and permission boundaries. Easy examples alone can make a low-cost route look more reliable than it will be in production.
Include the fallback bill
Suppose a hypothetical inexpensive attempt costs $0.01 and a stronger fallback costs $0.08. If 20% of attempts need that fallback, expected inference cost is $0.01 + 0.20 × $0.08 = $0.026 per attempt, before validation and other overhead.
That arithmetic assumes failures are detected and fallback finishes the task. Missed errors, repeated retries and manual intervention change the result. Compare cost per successful task at an acceptable quality level rather than reporting only the cheap model’s first-call cost.
Separate a router’s score from a guarantee
A router can use task type, input size or a measured validation outcome to choose a path. Any confidence signal must be calibrated on representative data. A model saying it is confident is not a substitute for checking a consequential result.
Escalate when inputs fall outside the evaluated scope. Keep authorization and irreversible-action controls consistent across model choices. A cheaper route should not gain permission to skip an approval or invent missing information.
Deploy routing as an experiment
OpenAI’s cost-optimization guidance discusses choosing an appropriate model as one lever among several. Test the route on your workload with the current models and prices; model capabilities and billing can change.
Record which path ran, its total usage, validation results and whether fallback occurred. Keep a rollback option and inspect quality changes as well as cost. Sometimes eliminating a repeated planning call saves more than routing that same unnecessary call to a smaller model.
Compare routing with a stronger-model baseline
Under the hypothetical costs above, and ignoring overhead, routing costs 0.01 + 0.08f dollars per attempt, where f is the fallback fraction. It costs less than an always-$0.08 path when f < 0.875. At an 87.5% fallback rate they are equal. This comparison assumes equivalent successful outcomes and a single fallback at most. Validation charges, undetected errors and slower completion can erase the apparent saving well before that boundary.