Most teams reduce inference costs by rewriting prompts or switching models manually. Both approaches require ongoing human judgment and break under volume. A prompt router solves this at the architecture level. It classifies each request before inference and sends it to the right model automatically.
The result: you stop paying GPT-4-class prices for tasks a smaller model handles correctly.
What a Prompt Router Actually Does
A prompt router sits between your application and your model endpoints. It receives an incoming request, evaluates it against a set of routing criteria, assigns it to a tier, and forwards it to the appropriate model.
It does not modify the prompt. It does not change your output format. It only decides which model processes the request.
The routing decision happens in milliseconds. The cost savings accumulate across every request.
The Four Routing Criteria
Before you write any routing logic, define your decision surface. Four criteria cover most production cases.
Task complexity. Is this a lookup, a classification, a summarization, or a multi-step reasoning task? Lookups and classifications rarely need a frontier model.
Token count. Input token length is a reliable proxy for complexity. A request under 300 tokens is usually lightweight. Over 2,000 tokens usually signals a complex task.
Latency requirement. Some requests are user-facing and need a response under 1 second. Others are batch jobs where 10 seconds is acceptable. Latency tolerance affects which models are eligible.
Cost threshold. Set a per-request cost ceiling per tier. For example: lightweight tier ≤ $0.001, standard tier ≤ $0.005, complex tier ≤ $0.02. These numbers force you to be explicit about acceptable spend per task type.
Document these four criteria before building anything. They become your routing spec.
Step-by-Step: Classify Requests Into Tiers
Three tiers cover most workloads.
- Lightweight: Simple extraction, yes/no classification, short-form lookup. Token count under 400. Latency-sensitive or high-volume.
- Standard: Summarization, structured output generation, moderate reasoning. Token count 400–1,500. Moderate latency tolerance.
- Complex: Multi-step reasoning, long-document analysis, code generation with context. Token count over 1,500 or explicit complexity signals.
To classify requests automatically, run a lightweight classifier before the main inference call. This classifier can be a small model, a rules-based function, or a combination.
A practical starting point is a rules-based function:
- Count input tokens.
- Check for explicit complexity signals in the request metadata (e.g., a
task_typefield your application already sets). - Apply a keyword list for known complex task types (e.g., "compare", "analyze", "generate code").
- Assign a tier based on the combined score.
This classifier should add no more than 20–50ms to your pipeline. If you use a model for classification, use the smallest available — a 7B-parameter model or a fine-tuned classifier is sufficient. Do not use a frontier model to decide whether to use a frontier model.
Log every classification decision. After two weeks, audit the logs. You will find misclassified requests. Adjust your rules or retrain your classifier based on real data.
Assign Model Targets Per Tier
Once tiers are defined, map each tier to a primary model.
Example mapping:
- Lightweight →
gpt-4o-minior equivalent small model - Standard →
gpt-4oor a mid-tier model - Complex →
gpt-4owith extended context, or a reasoning-optimized model
Do not hard-code model names as strings scattered across your codebase. Store the tier-to-model mapping in a single configuration file. When a model is deprecated or you want to swap providers, you change one file.
Build the Fallback Chain
Every tier needs a fallback. Model endpoints go down. Rate limits get hit. A router without a fallback chain becomes a single point of failure.
For each tier, define:
- Primary model — the default target.
- Secondary model — same tier, different provider or endpoint.
- Escalation model — one tier up, used only when both primary and secondary are unavailable.
Example fallback chain for the lightweight tier:
- Primary:
gpt-4o-mini - Secondary:
claude-haiku(or equivalent) - Escalation:
gpt-4o(standard tier model, used only as last resort)
The escalation step costs more. That is acceptable. The alternative is a failed request.
Set a timeout for each model attempt. If the primary does not respond in 800ms, move to secondary. If secondary does not respond in 1,200ms, escalate. Log every fallback event. A spike in fallback usage is an early signal that a primary endpoint is degrading.
What This Looks Like in Practice
A team running 50,000 requests per day with no routing sends everything to a frontier model. At $0.01 per request average, that is $500/day.
With a router that correctly classifies 60% of requests as lightweight and sends them to a small model at $0.0008 per request:
- 30,000 lightweight requests × $0.0008 = $24
- 20,000 standard/complex requests × $0.01 = $200
- Total: $224/day
That is a 55% cost reduction. The prompts did not change. The outputs for lightweight tasks are equivalent. The only change is where each request goes.
The Operational Discipline That Makes It Work
A router is not a one-time build. It requires:
- Weekly review of classification accuracy
- A feedback loop from output quality checks back to tier assignments
- Version control on the tier-to-model mapping (treat it like code)
- Alerts when fallback usage exceeds a threshold
Without this discipline, the router drifts. Tasks get misclassified. Costs creep back up. Quality degrades silently.
Building the router takes a day. Maintaining the feedback loop is what keeps it working.
If you are running AI workflows at volume and want to audit your current inference cost structure, start a conversation →