Routing

Also known as: router, intent routing, query routing, semantic routing, model routing (LLM routing), classifier-dispatch

Intent

Put a classifier in front of several specialised paths and run exactly one of them per input. Each path gets its own prompt, tools, model and guardrails, so tuning one kind of request no longer degrades the others. The router decides once and then steps aside: it never sees the answer and never decides again.

Problem

You are building the support assistant of an online shop that sells headphones and speakers. Customers write about four things: refunds, technical problems ("my earbuds stopped pairing after the last update"), delivery tracking, and general product questions.

The first version is one system prompt that covers all four, with every tool attached and the largest model behind it. Each fix for one topic causes trouble somewhere else:

Anthropic's "Building effective agents" describes exactly this: without routing, optimising for one kind of input can hurt performance on the others.

Four different kinds of customer messages all go into one support prompt that holds every rule and tool, and the answers suffer. I want my money back My earbuds won't pair Where is my parcel? What are your hours? One support prompt all rules, all tools refunds + troubleshooting + tracking + FAQ large model for every query Answers diagnostics on refunds stiffer tech answers big-model price for FAQs each fix breaks another
One prompt for every kind of request: each topic's rules and tools get in the way of the others, and easy questions cost as much as hard ones.

Solution

Split the assistant into specialised handlers and put a small, fast classification step in front of them. The router reads the message, picks one label from a closed list, and your code calls the handler for that label. Each handler is written and tested on its own.

  1. Classify. The customer writes "My earbuds stopped pairing with my laptop after the last update." The router, a small model with a structured-output schema, returns {"route": "tech_support", "confidence": 0.88}. It does not answer the customer.
  2. Apply the policy. Code checks the label against the fixed list and compares the confidence with a threshold, say 0.7. Above it, the route stands. Below it, or on none, the request goes to a fallback: a generalist handler, one clarifying question, or a human queue.
  3. Dispatch. Code, not the model, calls the tech-support handler: a large model with a troubleshooting prompt and the run_diagnostics and search_kb tools. The refund rules and refund tools are not in its context at all.
  4. Answer and log. The handler's reply goes straight back to the customer. The route, confidence and router version are logged so you can measure accuracy and spot drift later.

A refund question takes a different path with a strict prompt and an approval step, and "what are your opening hours?" goes to a small model with retrieval over the FAQ. The same split also cuts cost, because only the hard requests reach the large model.

Who decides the flow

The developer decides which paths exist. The router decides which path this request takes, once. The handler decides how to solve the task. Because the set of routes is fixed in code, routing is a workflow and not an agent, in Anthropic's sense of the terms.

The word "router" is also used for two other designs that hand control around differently. The table below is how this site separates them.

Three designs that are all called "routing" somewhere
PatternWho decidesWhenDoes control come back to the decider?
Router (this page)A dedicated classifier step: LLM, embeddings, small model or rulesOnce per requestNo. The handler's output is the answer.
SupervisorA manager agentEvery stepYes. Workers report back and the supervisor picks the next step.
HandoffThe currently active agentWhenever it judges a peer is better placedNo. The receiving agent takes over the conversation.

Products called a "triage agent" (OpenAI Agents SDK) or a "coordinator" (Google ADK) are mostly handoffs: a full agent transfers the conversation by calling a transfer_to_* tool, and later turns may stay with the specialist or move again.

Real-world analogy

A hospital triage desk. The triage nurse asks a few questions, assigns a category, and sends the patient to the emergency room, the general practitioner or the pharmacy. The nurse does not treat anyone and does not follow the patient into the consulting room. The departments are the handlers, the list of departments is the route set, and "I'm not sure, please wait for the duty doctor" is the fallback route.

The analogy stops at the edges. A nurse who sees a patient deteriorate in the waiting room will re-triage them. A plain router has no such second look: once the request is dispatched, a misroute stays a misroute unless the handler notices and you have built a way back.

Structure

A request goes to a router that returns a label and a confidence, a code policy dispatches it to one of four handlers or a fallback, the handler's answer goes straight to the caller, and the decision is logged. Request + history Router label + confidence closed enum Policy threshold in code Decision log route + confidence Refund large model, refund tools Tech support diagnostics tools Order status small model, orders API General FAQ small model + FAQ index Fallback clarify, generalist, human Answer to caller 1 2 3 4 5
The accent line is the path taken by the pairing question. Solid lines carry the request, dashed lines carry results and logs. No line returns to the router: it never sees the answer.
  1. Router. Maps the request (and maybe a short history window) to one label from the route registry, plus a confidence and sometimes extracted parameters such as an order ID. It can be an LLM with structured output, an embedding matcher, a small fine-tuned classifier or plain rules. The route descriptions are its documentation: selection quality depends on them.
  2. Decision policy. Thresholds, tie-breaking, the none route, budget limits. It lives in code, not in the prompt, so it can be tested and changed without touching the model.
  3. Handlers. The specialised paths. Each has its own prompt, tools, model, temperature and guardrails, and can be anything from a prompt template to a RAG index, a chain or a full agent.
  4. Fallback. Where low-confidence and out-of-scope requests go: a generalist handler, a single clarifying question, or a human queue.
  5. Decision log. Input hash, route, confidence, router version, latency and downstream outcome for each request. You need it for evaluation and to notice drift.

Prompt generator

Describe your routes and the generator writes two things: a system prompt for the router model (classification only, closed output) and the dispatch policy your code should apply to its answer. Use the prompt with any model that supports structured output, for example as the router node of a LangGraph graph or a CrewAI @router method.

Routes one row per route; a "none" route is added for you
Low confidence or none
Output format

Generated prompt

Highlighted text comes from the form. [Bracketed] text is still to fill in.

The route descriptions do most of the work

Router quality depends less on the instructions than on how distinct the route descriptions are and on examples for the pairs that get confused. Anthropic's ticket-routing guide reports that adding retrieved, labelled examples to the prompt raised accuracy from 71% to 93%; that figure comes from Anthropic's own cookbook and has not been independently replicated. Build a labelled test set before tuning the wording.

Applicability

Use it when

Avoid it when

How to implement

  1. Collect real inputs first. Pull a few hundred past messages or tickets and look at how people triage them today.
  2. Design the taxonomy. Routes must be distinct enough that two annotators agree. Merge routes whose handlers would be nearly the same. Anthropic's ticket-routing guide suggests a hierarchy (a top-level classifier, then sub-classifiers) once you pass about 20 intents, at the cost of an extra hop.
  3. Label an evaluation set. Cover every route and include deliberately ambiguous and out-of-scope messages labelled none.
  4. Pick the cheapest router that reaches your accuracy target. See the table below. Many systems stack them: rules first, embeddings next, an LLM only for the uncertain tail.
  5. Close the output. Use an enum or Literal type for the route so the model cannot invent names, and always include a none option. If you want a short reason to inform the label, put the reasoning field before the route field.
  6. Write the decision policy in code. Threshold, fallback, at most one clarifying question, timeouts with a default route, and logging.
  7. Build and test each handler on its own. This is the main payoff: changing the refund prompt can no longer break troubleshooting.
  8. Evaluate end to end, then monitor. Look at the confusion matrix, then at task success, cost and latency per route. In production, watch the route mix, the fallback rate and how often requests get rerouted.
Router implementations (latency figures from the sources named)
Router typeLatency and costGood atWeak at
Rules, keywords, metadataAbout zeroDeterministic, free, auditableParaphrase and other languages
Embedding similarity to example utterancesMillisecondsAdding a route is adding examplesNeeds threshold tuning; weak on negation and long multi-intent input
Small fine-tuned classifier (BERT-style)Milliseconds, self-hostedCalibrated scores, cheap at scaleNeeds labelled data and retraining when routes change
Small LLM trained for routingArch-Router (1.5B) reported about 51 msReads route descriptions and multi-turn contextYou have to host it
General LLM with structured outputOne extra call; the Arch-Router paper measured 0.8 to 1.5 s for GPT-4o and Claude 3.7 Sonnet as routersZero-shot, handles nuance, explains itselfSlowest, costs tokens, can be inconsistent

In frameworks, routing usually shows up as a conditional edge rather than a dedicated class. LangGraph uses add_conditional_edges with a structured-output router node (or Send for several routes). LlamaIndex has RouterQueryEngine with single or multi selectors. CrewAI Flows has @router plus @listen("label"). Aurelio's semantic-router matches against example utterances. OpenAI's Agents SDK and Google ADK express triage as handoffs instead.

Caveat. APIs move. semantic-router renamed its main classes and threshold parameters between 0.x and 1.x, and the claim that 1.x always picks the best route unless a threshold is set comes from search snippets of pages that no longer load (unverified). Agent Squad moved from awslabs to 2fastlabs. Check current docs before copying code.

Variants

Intent routing

The classic form described on this page: map a request to a specialised prompt, chain or agent. Anthropic's example splits general questions, refunds and technical support.

Query routing for RAG

Pick which index, retriever or tool answers a question. LlamaIndex's RouterQueryEngine can choose between a summary index for "summarise this" and a vector index for specific lookups. LangGraph's Adaptive RAG tutorial routes between a local vector store and web search.

Model routing

Choose which model answers, before any generation: easy queries to a small model, hard ones to a large one. RouteLLM (LMSYS) trains a predictor of whether the strong model would win on a query and routes to the weak model below a threshold. On its blog, LMSYS reports cost reductions against always using GPT-4 of over 85% on MT Bench, about 45% on MMLU and about 35% on GSM8K while keeping 95% of GPT-4's quality. Those are research numbers on 2024 models. A Microsoft community blog measured only about 4.5 to 14% savings with Foundry's model router on a 10-prompt test, which shows how much the result depends on the workload. Managed options include OpenRouter's Auto Router; vendor savings claims are self-reported.

Cascades

Run the cheap model first, let a scorer judge the answer, and escalate if it falls short. FrugalGPT and AutoMix work this way. Decisions are better informed because an answer exists, but an escalation costs extra generations. This is a hybrid with Evaluator–Optimizer, and RouterBench found cascades only beat the baselines while the judge's error rate stayed below about 0.1.

Hierarchical routing

A top-level classifier picks a broad category, then a sub-classifier refines it. Useful past about 20 intents (Anthropic's guide), at the cost of one hop per level. Not the same as hierarchical teams, where managers keep control.

Multi-label routing

The router picks several handlers, they run in parallel, and a synthesis step merges the results. LangChain's router pattern returns a list of Send objects for this. It overlaps with Parallelization, with the difference that the branches are chosen per input.

Tool routing

With hundreds of tools, select a subset before the agent sees them. Anthropic's Tool Search Tool defers most tool definitions and lets the model search them; Anthropic reports an 85% reduction in upfront tokens. Here the agent triggers the search, so it is a hybrid with the tool-use loop.

Route simple, supervise complex

AWS Bedrock's multi-agent collaboration has a "supervisor with routing" mode: simple requests go straight to one subagent, while complex or ambiguous ones fall back to full supervisor mode.

Pros and cons

Pros

  • Each handler can be tuned and tested in isolation, so a fix for one category does not regress the others.
  • Cheap requests stay cheap: only hard or risky ones reach the large model or the guarded path.
  • Easy to debug: each decision is one logged label with a confidence, drawn from a closed set.
  • Learned routers cost almost nothing to run. RouteLLM measured about $3 to $5 per million requests for most of its routers.
  • Risky categories can carry their own guardrails and approval gates.

Cons

  • Misroutes are silent. The handler usually does not know it got the wrong request and answers confidently.
  • An LLM router adds a call before every answer, from milliseconds for embeddings to over a second for a large model.
  • Routers often see only the latest message, so short follow-ups like "yes, do that" get misrouted.
  • The taxonomy needs upkeep. New products and new phrasing cause drift, and trained classifiers need retraining when routes change.
  • Savings from model routing depend heavily on the workload and are often smaller than vendor headlines.

Failure modes

Misclassification

A refund request lands in tech support, which runs diagnostics and answers with the wrong policy. Nothing errors, so you only see it in reroute rates or complaints.

Mitigation: a labelled evaluation set and a confusion matrix, few-shot examples for the pairs that get confused, retrieved similar examples in the prompt, and a hierarchy for large taxonomies.

Forced choice on out-of-scope input

The router always picks something, so "can you write me a poem?" ends up in the FAQ handler.

Mitigation: an explicit none or other route, a confidence threshold, and a default generalist handler.

Poorly calibrated confidence

An LLM router can report 0.9 confidence on requests it gets wrong, so a 0.7 threshold does not filter what you expect. The evidence for this is indirect: practitioners widely observe that self-reported confidence is poorly calibrated, and recommend not trusting it as is.

Mitigation: use token log-probabilities, agreement across several samples, or a classifier's probability, and calibrate the threshold on held-out data (semantic-router's fit, RouteLLM's calibrate_threshold).

Multi-intent and implicit requests

"Still waiting after two weeks, I want a refund" is read as only one of its two issues, or "still nothing in my letterbox" is not recognised as a delivery question. An angry tone gets classified as a complaint.

Mitigation: a priority rule in the prompt, examples of implicit requests, an instruction to focus on intent over emotion, multi-label routing if both issues must be handled, or one clarifying question.

Context-free follow-ups

"Yes, do that" is routed without knowing what "that" refers to.

Mitigation: pass a short history window to the router, use a conversation-aware router, or keep follow-ups with the current handler (handoff semantics).

Drift and route sprawl

A new product line or seasonal issue appears, accuracy decays, and new routes get added until their descriptions overlap.

Mitigation: monitor the route mix and the none rate, re-label samples regularly, merge similar routes, and treat any change to a route description as a model change that needs a regression run.

Benchmark-tuned model router fails on your traffic

RouteLLM's routers trained only on Chatbot Arena data were at about random level on MMLU and slightly below random on GSM8K.

Mitigation: calibrate and evaluate on your own traffic, and add in-domain labels; LMSYS reports that about 1,500 labelled MMLU samples helped substantially.

Adversarial rerouting

Attackers add text that pushes cheap-tier queries to the expensive model. In the "Rerouting LLM Routers" paper, white-box attacks upgraded 73 to 100% of weak-routed queries, the attack transferred to black-box settings, and perplexity filtering did not stop it.

Mitigation: treat routing as a security boundary when cost or permissions differ by route. Put rate limits and per-user budgets on expensive routes, watch for anomalies in the route mix, and never let the routing decision alone grant privileges.

Router as a single point of failure

A router outage or timeout blocks all traffic.

Mitigation: a timeout with a default route, and a cheap deterministic fallback router.

Relations with other patterns

Further reading