Routing
Also known as: router, intent routing, query routing, semantic routing, model routing (LLM routing), classifier-dispatch
Intent
Put a classifier in front of several specialised paths and run exactly one of them per input. Each path gets its own prompt, tools, model and guardrails, so tuning one kind of request no longer degrades the others. The router decides once and then steps aside: it never sees the answer and never decides again.
Problem
You are building the support assistant of an online shop that sells headphones and speakers. Customers write about four things: refunds, technical problems ("my earbuds stopped pairing after the last update"), delivery tracking, and general product questions.
The first version is one system prompt that covers all four, with every tool attached and the largest model behind it. Each fix for one topic causes trouble somewhere else:
- Rules leak across topics. You make the refund instructions strict (always check the order date, never promise money), and the troubleshooting answers turn stiff and defensive.
- The wrong tools get called. With diagnostics, refund and order tools all in one prompt, the model runs diagnostics on a refund question.
- Every query pays the top price. "What are your opening hours?" goes to the same frontier model as a firmware bug. RouteLLM's paper prices GPT-4 at about $24.7 per million tokens against $0.24 for Mixtral 8x7B, a gap of roughly 100x.
- The prompt only grows. More topics mean more instructions and tool definitions, so latency and cost rise for all queries, including the easy ones.
Anthropic's "Building effective agents" describes exactly this: without routing, optimising for one kind of input can hurt performance on the others.
Solution
Split the assistant into specialised handlers and put a small, fast classification step in front of them. The router reads the message, picks one label from a closed list, and your code calls the handler for that label. Each handler is written and tested on its own.
- Classify. The customer writes "My earbuds stopped pairing with my laptop after the last update." The router, a small model with a structured-output schema, returns
{"route": "tech_support", "confidence": 0.88}. It does not answer the customer. - Apply the policy. Code checks the label against the fixed list and compares the confidence with a threshold, say 0.7. Above it, the route stands. Below it, or on
none, the request goes to a fallback: a generalist handler, one clarifying question, or a human queue. - Dispatch. Code, not the model, calls the tech-support handler: a large model with a troubleshooting prompt and the
run_diagnosticsandsearch_kbtools. The refund rules and refund tools are not in its context at all. - Answer and log. The handler's reply goes straight back to the customer. The route, confidence and router version are logged so you can measure accuracy and spot drift later.
A refund question takes a different path with a strict prompt and an approval step, and "what are your opening hours?" goes to a small model with retrieval over the FAQ. The same split also cuts cost, because only the hard requests reach the large model.
Who decides the flow
The developer decides which paths exist. The router decides which path this request takes, once. The handler decides how to solve the task. Because the set of routes is fixed in code, routing is a workflow and not an agent, in Anthropic's sense of the terms.
The word "router" is also used for two other designs that hand control around differently. The table below is how this site separates them.
| Pattern | Who decides | When | Does control come back to the decider? |
|---|---|---|---|
| Router (this page) | A dedicated classifier step: LLM, embeddings, small model or rules | Once per request | No. The handler's output is the answer. |
| Supervisor | A manager agent | Every step | Yes. Workers report back and the supervisor picks the next step. |
| Handoff | The currently active agent | Whenever it judges a peer is better placed | No. The receiving agent takes over the conversation. |
Products called a "triage agent" (OpenAI Agents SDK) or a "coordinator" (Google ADK) are mostly handoffs: a full agent transfers the conversation by calling a transfer_to_* tool, and later turns may stay with the specialist or move again.
Real-world analogy
A hospital triage desk. The triage nurse asks a few questions, assigns a category, and sends the patient to the emergency room, the general practitioner or the pharmacy. The nurse does not treat anyone and does not follow the patient into the consulting room. The departments are the handlers, the list of departments is the route set, and "I'm not sure, please wait for the duty doctor" is the fallback route.
The analogy stops at the edges. A nurse who sees a patient deteriorate in the waiting room will re-triage them. A plain router has no such second look: once the request is dispatched, a misroute stays a misroute unless the handler notices and you have built a way back.
Structure
- Router. Maps the request (and maybe a short history window) to one label from the route registry, plus a confidence and sometimes extracted parameters such as an order ID. It can be an LLM with structured output, an embedding matcher, a small fine-tuned classifier or plain rules. The route descriptions are its documentation: selection quality depends on them.
- Decision policy. Thresholds, tie-breaking, the
noneroute, budget limits. It lives in code, not in the prompt, so it can be tested and changed without touching the model. - Handlers. The specialised paths. Each has its own prompt, tools, model, temperature and guardrails, and can be anything from a prompt template to a RAG index, a chain or a full agent.
- Fallback. Where low-confidence and out-of-scope requests go: a generalist handler, a single clarifying question, or a human queue.
- Decision log. Input hash, route, confidence, router version, latency and downstream outcome for each request. You need it for evaluation and to notice drift.
Prompt generator
Describe your routes and the generator writes two things: a system prompt for the router model (classification only, closed output) and the dispatch policy your code should apply to its answer. Use the prompt with any model that supports structured output, for example as the router node of a LangGraph graph or a CrewAI @router method.
Generated prompt
Highlighted text comes from the form. [Bracketed] text is still to fill in.
Router quality depends less on the instructions than on how distinct the route descriptions are and on examples for the pairs that get confused. Anthropic's ticket-routing guide reports that adding retrieved, labelled examples to the prompt raised accuracy from 71% to 93%; that figure comes from Anthropic's own cookbook and has not been independently replicated. Build a labelled test set before tuning the wording.
Applicability
Use it when
- Inputs fall into distinct categories that need different handling. Different prompts, tools, models or policies, and a human annotator would agree on the label.
- You can classify accurately. Anthropic names this as a precondition. Measure it on real traffic before you build the handlers.
- Difficulty varies and model prices differ a lot. Model routing sends easy queries to a cheap model. Gains depend on the candidate models having complementary strengths.
- You have several data sources or many tools. Querying all of them is slow and noisy; picking one index or a subset of tools is cheaper.
- You want to isolate risk. Refund requests can go to a tightly guarded path with an approval gate (Human-in-the-Loop) while FAQ answers stay fast.
Avoid it when
- Categories overlap or most requests mix several intents. Use a single capable agent, multi-label routing, or a Supervisor.
- The right specialist only becomes clear mid-conversation. Use handoffs or a supervisor that re-decides each turn. LangChain's docs warn that a stateful router carrying full history is hard to get right.
- The task must be split into subtasks you cannot list in advance. Use Orchestrator–Workers or Plan-and-Execute; routing only chooses among predefined paths.
- One handler would do. A single strong model with a good prompt can beat a router plus three specialists once you count misroutes, latency and maintenance.
- Traffic is uniform in difficulty. Model-routing savings shrink when there are few easy queries to send to the cheap model.
How to implement
- Collect real inputs first. Pull a few hundred past messages or tickets and look at how people triage them today.
- Design the taxonomy. Routes must be distinct enough that two annotators agree. Merge routes whose handlers would be nearly the same. Anthropic's ticket-routing guide suggests a hierarchy (a top-level classifier, then sub-classifiers) once you pass about 20 intents, at the cost of an extra hop.
- Label an evaluation set. Cover every route and include deliberately ambiguous and out-of-scope messages labelled
none. - Pick the cheapest router that reaches your accuracy target. See the table below. Many systems stack them: rules first, embeddings next, an LLM only for the uncertain tail.
- Close the output. Use an enum or
Literaltype for the route so the model cannot invent names, and always include anoneoption. If you want a short reason to inform the label, put the reasoning field before the route field. - Write the decision policy in code. Threshold, fallback, at most one clarifying question, timeouts with a default route, and logging.
- Build and test each handler on its own. This is the main payoff: changing the refund prompt can no longer break troubleshooting.
- Evaluate end to end, then monitor. Look at the confusion matrix, then at task success, cost and latency per route. In production, watch the route mix, the fallback rate and how often requests get rerouted.
| Router type | Latency and cost | Good at | Weak at |
|---|---|---|---|
| Rules, keywords, metadata | About zero | Deterministic, free, auditable | Paraphrase and other languages |
| Embedding similarity to example utterances | Milliseconds | Adding a route is adding examples | Needs threshold tuning; weak on negation and long multi-intent input |
| Small fine-tuned classifier (BERT-style) | Milliseconds, self-hosted | Calibrated scores, cheap at scale | Needs labelled data and retraining when routes change |
| Small LLM trained for routing | Arch-Router (1.5B) reported about 51 ms | Reads route descriptions and multi-turn context | You have to host it |
| General LLM with structured output | One extra call; the Arch-Router paper measured 0.8 to 1.5 s for GPT-4o and Claude 3.7 Sonnet as routers | Zero-shot, handles nuance, explains itself | Slowest, costs tokens, can be inconsistent |
In frameworks, routing usually shows up as a conditional edge rather than a dedicated class. LangGraph uses add_conditional_edges with a structured-output router node (or Send for several routes). LlamaIndex has RouterQueryEngine with single or multi selectors. CrewAI Flows has @router plus @listen("label"). Aurelio's semantic-router matches against example utterances. OpenAI's Agents SDK and Google ADK express triage as handoffs instead.
Caveat. APIs move. semantic-router renamed its main classes and threshold parameters between 0.x and 1.x, and the claim that 1.x always picks the best route unless a threshold is set comes from search snippets of pages that no longer load (unverified). Agent Squad moved from awslabs to 2fastlabs. Check current docs before copying code.
Variants
Intent routing
The classic form described on this page: map a request to a specialised prompt, chain or agent. Anthropic's example splits general questions, refunds and technical support.
Query routing for RAG
Pick which index, retriever or tool answers a question. LlamaIndex's RouterQueryEngine can choose between a summary index for "summarise this" and a vector index for specific lookups. LangGraph's Adaptive RAG tutorial routes between a local vector store and web search.
Model routing
Choose which model answers, before any generation: easy queries to a small model, hard ones to a large one. RouteLLM (LMSYS) trains a predictor of whether the strong model would win on a query and routes to the weak model below a threshold. On its blog, LMSYS reports cost reductions against always using GPT-4 of over 85% on MT Bench, about 45% on MMLU and about 35% on GSM8K while keeping 95% of GPT-4's quality. Those are research numbers on 2024 models. A Microsoft community blog measured only about 4.5 to 14% savings with Foundry's model router on a 10-prompt test, which shows how much the result depends on the workload. Managed options include OpenRouter's Auto Router; vendor savings claims are self-reported.
Cascades
Run the cheap model first, let a scorer judge the answer, and escalate if it falls short. FrugalGPT and AutoMix work this way. Decisions are better informed because an answer exists, but an escalation costs extra generations. This is a hybrid with Evaluator–Optimizer, and RouterBench found cascades only beat the baselines while the judge's error rate stayed below about 0.1.
Hierarchical routing
A top-level classifier picks a broad category, then a sub-classifier refines it. Useful past about 20 intents (Anthropic's guide), at the cost of one hop per level. Not the same as hierarchical teams, where managers keep control.
Multi-label routing
The router picks several handlers, they run in parallel, and a synthesis step merges the results. LangChain's router pattern returns a list of Send objects for this. It overlaps with Parallelization, with the difference that the branches are chosen per input.
Tool routing
With hundreds of tools, select a subset before the agent sees them. Anthropic's Tool Search Tool defers most tool definitions and lets the model search them; Anthropic reports an 85% reduction in upfront tokens. Here the agent triggers the search, so it is a hybrid with the tool-use loop.
Route simple, supervise complex
AWS Bedrock's multi-agent collaboration has a "supervisor with routing" mode: simple requests go straight to one subagent, while complex or ambiguous ones fall back to full supervisor mode.
Pros and cons
Pros
- Each handler can be tuned and tested in isolation, so a fix for one category does not regress the others.
- Cheap requests stay cheap: only hard or risky ones reach the large model or the guarded path.
- Easy to debug: each decision is one logged label with a confidence, drawn from a closed set.
- Learned routers cost almost nothing to run. RouteLLM measured about $3 to $5 per million requests for most of its routers.
- Risky categories can carry their own guardrails and approval gates.
Cons
- Misroutes are silent. The handler usually does not know it got the wrong request and answers confidently.
- An LLM router adds a call before every answer, from milliseconds for embeddings to over a second for a large model.
- Routers often see only the latest message, so short follow-ups like "yes, do that" get misrouted.
- The taxonomy needs upkeep. New products and new phrasing cause drift, and trained classifiers need retraining when routes change.
- Savings from model routing depend heavily on the workload and are often smaller than vendor headlines.
Failure modes
- Misclassification
-
A refund request lands in tech support, which runs diagnostics and answers with the wrong policy. Nothing errors, so you only see it in reroute rates or complaints.
Mitigation: a labelled evaluation set and a confusion matrix, few-shot examples for the pairs that get confused, retrieved similar examples in the prompt, and a hierarchy for large taxonomies.
- Forced choice on out-of-scope input
-
The router always picks something, so "can you write me a poem?" ends up in the FAQ handler.
Mitigation: an explicit
noneorotherroute, a confidence threshold, and a default generalist handler. - Poorly calibrated confidence
-
An LLM router can report 0.9 confidence on requests it gets wrong, so a 0.7 threshold does not filter what you expect. The evidence for this is indirect: practitioners widely observe that self-reported confidence is poorly calibrated, and recommend not trusting it as is.
Mitigation: use token log-probabilities, agreement across several samples, or a classifier's probability, and calibrate the threshold on held-out data (semantic-router's
fit, RouteLLM'scalibrate_threshold). - Multi-intent and implicit requests
-
"Still waiting after two weeks, I want a refund" is read as only one of its two issues, or "still nothing in my letterbox" is not recognised as a delivery question. An angry tone gets classified as a complaint.
Mitigation: a priority rule in the prompt, examples of implicit requests, an instruction to focus on intent over emotion, multi-label routing if both issues must be handled, or one clarifying question.
- Context-free follow-ups
-
"Yes, do that" is routed without knowing what "that" refers to.
Mitigation: pass a short history window to the router, use a conversation-aware router, or keep follow-ups with the current handler (handoff semantics).
- Drift and route sprawl
-
A new product line or seasonal issue appears, accuracy decays, and new routes get added until their descriptions overlap.
Mitigation: monitor the route mix and the
nonerate, re-label samples regularly, merge similar routes, and treat any change to a route description as a model change that needs a regression run. - Benchmark-tuned model router fails on your traffic
-
RouteLLM's routers trained only on Chatbot Arena data were at about random level on MMLU and slightly below random on GSM8K.
Mitigation: calibrate and evaluate on your own traffic, and add in-domain labels; LMSYS reports that about 1,500 labelled MMLU samples helped substantially.
- Adversarial rerouting
-
Attackers add text that pushes cheap-tier queries to the expensive model. In the "Rerouting LLM Routers" paper, white-box attacks upgraded 73 to 100% of weak-routed queries, the attack transferred to black-box settings, and perplexity filtering did not stop it.
Mitigation: treat routing as a security boundary when cost or permissions differ by route. Put rate limits and per-user budgets on expensive routes, watch for anomalies in the route mix, and never let the routing decision alone grant privileges.
- Router as a single point of failure
-
A router outage or timeout blocks all traffic.
Mitigation: a timeout with a default route, and a cheap deterministic fallback router.
Relations with other patterns
- Prompt Chaining: a router is often the first step in front of several chains. A gate with two outcomes inside a chain is a tiny router.
- Parallelization: runs several branches chosen in advance by the developer. Multi-label routing sits in between, with a classifier choosing the branches per input.
- Evaluator–Optimizer: cascades borrow its evaluator to decide whether to escalate. Routing proper decides before any answer exists.
- ReAct (tool-use loop): an agent effectively routes on every turn when it picks a tool. Tool routing narrows its tool set.
- Plan-and-Execute: a router in front of a planner can send trivial requests to a cheap handler and skip planning.
- Orchestrator–Workers: the orchestrator invents subtasks at run time and merges their results. A router only picks among predefined paths and never sees the result.
- Supervisor: the pattern most often confused with routing. A supervisor re-decides after every worker step and receives the results; a router decides once.
- Hierarchical teams: hierarchical routing is a tree of classifiers that each decide once; hierarchical teams are trees of managers that keep control.
- Swarm / Handoffs: a "triage agent" that calls
transfer_to_<agent>is a handoff. The receiving agent keeps the conversation and may transfer again. - Group Chat: its speaker selector is a router that fires on every message over a shared transcript.
- Blackboard: its control component also picks who acts next, but it reads a shared workspace repeatedly instead of classifying one request.
- Council / Debate: the opposite choice. A council sends the same question to several members on purpose; a router sends it to exactly one.
- Human-in-the-Loop: a human queue is a natural fallback route, and risky routes can carry approval gates.
Further reading
- Building effective agents (Anthropic): the definition of the routing workflow and when it fits.
- Ticket routing (Anthropic docs): a full LLM router for support tickets, with prompt structure, success targets and evaluation.
- Router (LangChain docs): stateless and stateful routers, multi-route fan-out, and when to switch to handoffs or subagents.
- RouteLLM: Learning to Route LLMs with Preference Data (Ong et al., LMSYS / UC Berkeley): model routing, the cost-quality metrics and how routers generalise.
- Doing More with Less: A Survey on Routing Strategies (Varangot-Reille et al.): pre-generation routing versus cascades, and the router families.
- Rerouting LLM Routers (Shafran et al.): how routers can be manipulated, and why routing is a security boundary.