Prompt Chaining
Also known as: pipeline, sequential workflow, chain, sequential orchestration (Microsoft), linear delegation (Azure Architecture Center), deterministic flow (OpenAI Agents SDK), SequentialAgent or sequential process (Google ADK, CrewAI)
Intent
Split one hard task into a fixed sequence of smaller LLM calls and plain code steps. Each step gets a narrow instruction and the output of the step before it, and code can check the result between steps before anything moves on.
The order of the steps is written by the developer. The model never decides what runs next.
Problem
A company wants a French risk memo for every English supplier contract it receives. The legal team reads French; the contracts run to about 40 pages.
The first version is one prompt: "read this contract, find the risky clauses, rate them, write a memo in French following our template". It works on some contracts. On others it breaks in ways that are hard to see:
- Steps get skipped. The model holds five goals at once and drops some of them. Clauses go missing from the memo and nobody notices.
- Made-up details. Clause numbers appear that are not in the contract, and there is no intermediate list to check them against.
- Instructions lost in a long prompt. Some memos come out in English because the language rule sits at the end of a long prompt.
- No way to find the faulty part. When a memo is wrong, you cannot tell whether extraction, rating, writing or translation failed. The only test available is end to end.
- One configuration for everything. Translation would be fine on a small, cheap model and risk rating needs the strongest one, but a single call has a single model.
- The process gets faked. If you ask one prompt to "draft, critique, then refine", the model may write a weak draft on purpose and then "fix" it. Sun et al. (2024) call this simulated refinement, and present it as a likely explanation rather than a proven one.
Solution
Cut the task where the type of work changes, give each piece its own short prompt, and run the pieces in an order fixed in code. Between steps, code parses the output and checks it. A check that fails can retry the step, stop the chain, or hand the case to a person. Anthropic's "Building effective agents" calls these checks gates and presents prompt chaining as the first and simplest workflow.
For the contract memo, the chain has five steps:
- Extract. A small model lists the clauses as JSON:
[{id, title, text}]. Gate: the JSON parses and holds at least one clause. - Classify. A strong model rates each clause low, medium or high against a rubric. Gate: every clause id from step 1 is present and every rating is one of the three values.
- Draft. The memo is written in English from the high and medium items only.
- Translate. A cheap model translates it into French. Gate: a language detector says French.
- Format. Plain code puts the text into the legal template. No LLM call.
Each step is small enough to prompt well, test on its own and debug. When a memo is wrong, the stored intermediate results show which step failed.
Who decides the flow: the developer, at design time. The model writes the content of each step, and at most produces a value that a gate reads. It never picks the next step. That is what makes this a workflow and not an agent. In Plan-and-Execute an LLM writes the step list at run time; in Orchestrator–Workers a lead model invents the subtasks per request. Here the list is in your code.
Real-world analogy
A newspaper desk. The reporter files a story, the fact-checker checks names and figures, the copy editor fixes style, the translator produces the foreign edition, and layout puts it on the page. The desk order is set by the managing editor, not by the people at each desk. The fact-checker is a gate: a story with a wrong figure goes back to the reporter instead of moving on.
The analogy stops at judgement. A human copy editor who notices that a paragraph is missing will say so. An LLM step usually works with whatever it is handed and carries on, which is why a chain needs explicit gates and why each step should see the original source when it needs ground truth.
Structure
- Step. One focused transformation: a prompt template, a model and its settings, and a parser that turns raw text into a typed value. A step can also be plain code or a whole tool-using agent.
- Gate. Code, or a cheap classifier call read by code, that checks the parsed value: schema, counts, a regex, a business rule. It returns pass, retry, repair, early exit or escalate.
- Handoff contract. Exactly what the next step receives: the previous output, and the original input or earlier results only when needed. Too little loses information; too much bloats the prompt.
- Code step. Deterministic work such as formatting, a database lookup or running tests. Cheaper and easier to test than an LLM call.
- Early exit. A controlled stop when a gate keeps failing or the input is out of scope: an error, a fallback answer, or a person.
- State and trace. Intermediate results stored by name, so runs can be inspected, cached and resumed, with one trace span per step (prompt, output, tokens, latency, gate verdict). The runner that calls the steps in order is often just a
forloop.
Prompt generator
Describe the goal and list the steps in order. For each step give its single job, its output format and the check that runs after it. Set the model to code for steps that need no LLM. The result has two parts: rules for the runner code (retries, early exit, budget, tracing), then one prompt per LLM step with its input tags. Paste the step prompts into your own runner, a LangGraph node, or any framework's sequential steps.
Generated prompt
Highlighted text comes from the form. [Bracketed] text is still to fill in.
Most chain failures happen at handoffs: a malformed JSON, a dropped clause, a field the next step needed. Put a cheap code check right after extraction and planning steps, because errors there spread to everything downstream. Keep a single-prompt baseline and keep only the steps that beat it on your eval set.
Applicability
Use it when
- The task splits into fixed steps known in advance. This is Anthropic's main criterion. The order must not depend on what you find along the way.
- You need to inspect intermediate results. For audit, compliance or debugging. Claude's current prompting docs say explicit chaining stays useful even with strong reasoning models when you need to inspect intermediate outputs or enforce a pipeline structure.
- Steps need different models, settings or tools. A small model for extraction and translation, a strong one for the risk rating.
- Some intermediate results can be checked by code. Schemas, counts, tests, language detection. These checks are where the reliability comes from.
- Predictability matters and some extra latency is acceptable. Anthropic describes the trade as latency for accuracy. Users and regulators get the same path every time.
Avoid it when
- One well-written prompt already passes your eval. Every extra step adds a round trip and tokens. Keep the single call.
- The steps do not depend on each other. Run them at the same time with Parallelization.
- The kind of input decides which pipeline to run. Put Routing in front of several chains.
- You need "repeat until good enough". That is a loop with an explicit cap: Evaluator–Optimizer.
- The number and kind of steps depend on what is found. Open-ended research or multi-file coding call for ReAct, Plan-and-Execute or Orchestrator–Workers.
- Steps must go back to earlier stages or discuss. The Azure guide lists backtracking, dynamic routing and collaboration as reasons to avoid sequential orchestration. Use a graph with back edges, Swarm / Handoffs or Group Chat.
- Latency is critical, as in real-time chat. Use fewer, larger calls, parallel steps, or stream only the last step.
The boundary moves both ways. If you keep adding branches for unforeseen cases, or gates fail often because inputs vary more than the steps assume, one step (or the whole chain) should become an agent. A common middle ground keeps the outer chain and makes one step an agent. If traces show an agent takes the same path almost every time, fix that path in code: OpenAI's Agents SDK docs say orchestrating with code is more predictable in speed, cost and performance.
How to implement
- Write the single-prompt baseline first. The chain must beat it on an eval set to justify its cost.
- Find the seams. Good split points are where the output type changes (text to data to prose), where a different skill is needed (extraction, reasoning, style, translation), where correctness is cheap to check, or where a different model fits better.
- Define a data contract for each handoff. Use structured output (JSON schema, Pydantic) for anything code reads. Use free text only when the next consumer is another LLM, and then wrap each upstream result in its own XML tag (
<outline>,<draft>) so the next prompt can refer to it by name. - Write one narrow prompt per step. State its single goal, input format and output format, and what not to do ("do not translate yet").
- Add gates where errors are cheap to detect and costly to pass on. Right after extraction and planning steps. On failure, retry with the error message fed back.
- Set retry and stop rules. A maximum number of retries per step, a total token or cost budget, and a timeout per call and per chain. Keep transport retries (rate limits) separate from retries for bad output.
- Choose a model per step. Small models for classification, extraction and formatting, strong ones for reasoning and synthesis (Azure Architecture Center cost guidance).
- Trace and test each step on its own, then end to end. Persist state after each step so a crash resumes instead of restarting.
- Cache deterministic or repeated steps by step name, prompt version, model and input hash.
A framework is optional. Anthropic's cookbook implements the whole pattern as a loop that appends the previous result to the next prompt (its example turns a messy performance report into a markdown table in four steps), and Anthropic warns that framework abstractions can hide prompts and responses. Agent frameworks package the same idea as "sequential orchestration of agents"; if a step needs no tools or multiple turns, a plain LLM call is cheaper and easier to test.
| Framework | How this pattern appears |
|---|---|
| Raw API | A for loop over prompts; gates are your own if statements. |
| LangChain (LCEL) | prompt | model | parser builds a RunnableSequence; .with_retry(), .with_fallbacks(), with_structured_output. Streams end to end only if every component supports it. |
| LangGraph | StateGraph with add_edge for the line and add_conditional_edges for a gate; typed shared state. |
| OpenAI Agents SDK | Several Runner.run calls in code, a checker agent with output_type, an if for the gate, all inside trace(). |
| Google ADK | SequentialAgent(sub_agents=[...]) with output_key and {key} templating. Now marked deprecated in favour of the ADK 2.0 graph Workflow. |
| CrewAI | Process.sequential; context=[...] picks earlier tasks; guardrail with guardrail_max_retries (default 3). |
| Semantic Kernel | SequentialOrchestration(members=[...]); labelled experimental, not in the Java SDK. |
| Microsoft Agent Framework | SequentialBuilder; passes the full conversation by default, chain_only_agent_responses=True passes only the previous response; cannot go back to an earlier agent. |
| DSPy | A module whose forward() calls sub-modules in order; a compiler tunes every step's prompts against a metric. |
| LlamaIndex Workflows | @step methods linked by typed events, from StartEvent to StopEvent. |
Caveat. These APIs move fast. Google ADK's docs mark SequentialAgent deprecated without a version or removal date, Semantic Kernel's agent orchestration is experimental and Microsoft Agent Framework appears to be its successor, and DSPy's assertion API has changed between versions (current names not verified). Graph frameworks are the usual upgrade path: a chain is a linear graph, a conditional edge turns it into routing, and a back edge turns it into a loop.
Variants
Linear chain
A, then B, then C, with no checks. The cookbook chain() loop or an LCEL pipe. Fine for prototypes and low-stakes transforms.
Gated chain
Code checks between steps decide pass, retry or early exit. OpenAI's deterministic-flow example simply stops when the outline checker says the outline is poor or off-genre. A gate that stops the run with a clear reason is doing its job.
Hybrid chain (LLM and code steps)
Some steps are parsing, retrieval, database lookups, running tests or formatting. Decomposed Prompting (Khot et al., 2023) lets a sub-task be a prompt, a trained model or a symbolic function. The Format step in the contract example is one.
Extract, transform, generate
Unstructured input becomes structured data, then processed data, then prose. Document pipelines take this shape. DocETL rewrites such pipelines into finer chains and reports 25 to 80% more accurate outputs than well-engineered baselines on four tasks (Shankar et al., 2024, self-reported).
Draft, critique, refine
A fixed three-step chain: generate, review against criteria, rewrite once. Claude's prompting docs name it the most common chaining pattern. If the refine step loops until a criterion passes, it becomes Evaluator–Optimizer.
Chain with embedded loops ("flow engineering")
A mostly linear pipeline where some stages iterate against tests. AlphaCodium runs a linear pre-processing chain (reflect on the problem, reason about the tests, propose and rank solutions, generate extra tests), then fixes code against tests. The fix loop itself is an evaluator–optimizer loop.
Per-item chain
The same chain runs on each document or clause, often in parallel, then a reduce step merges the results. The fan-out belongs to Parallelization; each lane is a chain.
Chain with human checkpoints
A pause after a step for approval or edits, for example before the memo is sent. Microsoft Agent Framework has with_request_info; LangGraph has interrupts. See Human-in-the-Loop.
Chain of agents
Each step is a full agent with tools and its own loop, and the order between agents stays fixed. This is what Semantic Kernel, ADK and CrewAI call sequential orchestration. Inside each step, see ReAct.
Compiled chain
The prompts and few-shot examples of every step are tuned automatically against a metric. DSPy does this with a compiler over modules composed in forward().
Generated chain
An LLM writes the step list once, then code runs it as a fixed chain. That is Plan-and-Execute.
Pros and cons
Pros
- Usually more accurate, because each call is an easier task. AlphaCodium (a multi-stage flow, self-reported) raised GPT-4's pass@5 on its CodeContests validation set from 19% to 44%.
- Every intermediate result can be logged, unit-tested and replayed. In the AI Chains user study (Wu et al., 2022, 20 participants), people debugged by testing individual steps and rated chaining higher for transparency and control.
- Each step can use its own model, temperature and tools, so most steps can run on a cheap model.
- The same path runs every time, which makes cost and latency predictable and audits possible.
- Steps can be cached, and long shared prefixes can use prompt caching.
- No framework needed: the runner is a few lines of code.
Cons
- Higher latency: the sum of every step plus parsing and retries, with no parallelism inside a strict chain.
- Usually more tokens overall: repeated instructions, repeated context, retries. AlphaCodium uses about 15 to 20 LLM calls per solution.
- Errors compound unless gates catch them. As arithmetic, not a measurement: if each of 5 independent steps succeeds 95% of the time, the chain succeeds about 77% of the time; with 10 steps, about 60%.
- Rigid: inputs that do not fit the assumed steps produce garbage.
- Steps are coupled: changing one step's output format can break the next step's parser.
The compounding arithmetic assumes independent failures, and real failures are correlated. It also cuts both ways: each smaller step is easier, so its success rate is higher than the single prompt's, and a gate with retries raises it further. The goal is steps that are easy and checkable, not the fewest steps.
Most of the evidence comes from models released before built-in extended thinking (LaMDA in AI Chains, GPT-4 in AlphaCodium and Sun et al.). The sources include no recent controlled comparison of a chain against a single call to a reasoning model, so treat the size of the gains as model-dependent and measure on your own task.
Failure modes
- Cascading errors
-
A wrong extraction early on poisons every later step. In the contract chain, a clause missed in step 1 never reaches the memo, and the memo still reads well.
Mitigation: gates right after early steps; checks against known-good data (AlphaCodium's "test anchors" stop a wrong generated test from undoing a fix that passed known tests); let a person fix intermediate outputs.
- Information loss
-
A later step needs a detail that an earlier step dropped or summarized away, like a game of telephone. The draft step knows the risk rating but not the exact clause wording.
Mitigation: pass the original input or the earlier result along when a step needs ground truth; store large artifacts and pass references; test with inputs that contain rare details.
- Context bloat
-
Passing the full history to every step makes prompts long, slow, costly and distracting.
Mitigation: minimal handoff contracts; summarize or prune between steps. Sources disagree here: Cognition argues that when steps are long-running agents making implicit choices, narrow handoffs lose those choices. For short, well-specified LLM calls, minimal contracts work well.
- Format drift and parse failures
-
The model wraps the JSON in prose or drops a field, and the next step crashes or, worse, carries on with half the data.
Mitigation: native structured output, schema validation with the error fed back on retry (as Instructor does), strict parsers that fail loudly, YAML instead of JSON when the output contains code (AlphaCodium's choice: no quote escaping, fewer tokens).
- Miscalibrated gates
-
Too strict and you get endless retries and false rejections. Too loose and errors slip through. An LLM-judge gate has its own error rate.
Mitigation: prefer code checks; measure each gate's false-pass and false-reject rates on labelled examples; cap retries; log every verdict and review rejected cases.
- Over-decomposition
-
Many tiny steps add latency and cost, and every handoff is a chance to lose information.
Mitigation: merge steps that never fail on their own. Keep a step only if it improves an eval metric or makes a check possible.
- Simulated process
-
One prompt asked to "draft, then critique, then refine" goes through the motions. Sun et al. (2024) found three separate prompts beat one combined prompt for summarization, though the combined prompt wrote better critiques, so separation seems to matter most for the generation steps.
Mitigation: make each phase a separate call.
- Inputs that do not fit the steps
-
A document with no clauses section, or a contract in German, goes through the chain anyway and produces nonsense.
Mitigation: an input classifier gate at the start, with early exit or Routing to another chain.
- Silent regressions
-
Editing one prompt changes its output distribution and quietly breaks a later step.
Mitigation: version prompts and contracts; per-step assertion tests on real samples, rerun on every pipeline change; an end-to-end eval on every change.
- Retry storms
-
Retries on every step multiply calls, and transport retries stack on top of validation retries.
Mitigation: per-step and global budgets for tokens, time and money; one retry layer for transport errors, not two (turn off the SDK's retries if a wrapper already retries).
- Hidden errors
-
A step returns an empty or placeholder value and later steps carry on.
Mitigation: surface errors instead of hiding them; make "unknown" an explicit field in contracts; gates that reject empty values.
- Injected instructions travelling down the chain
-
Text in the contract (or a tool result) contains instructions, and a later step obeys them.
Mitigation: wrap upstream outputs in tags and tell each step they are data; apply content checks at several points, not only at the end; give each step the fewest tools it needs.
- Interrupted long chains
-
A crash or deploy mid-chain loses all the work done so far.
Mitigation: persist state after each step (LangGraph or Agent Framework checkpointers), make steps idempotent, resume from the last completed step.
Relations with other patterns
- Routing: a classifier picks which chain, prompt or model runs, and often sits in front of several chains. A gate that continues or exits is a tiny router, but a chain's order never depends on the input type.
- Parallelization: runs independent steps at the same time, where a chain runs dependent steps one after another. They combine often: map a chain over many items in parallel, then reduce.
- Evaluator–Optimizer: a generate and evaluate loop that repeats until a criterion passes. Draft, critique, refine is one unrolled pass of it, and a capped gate retry is the smallest step toward it.
- ReAct (tool-use loop): the model picks the next action every turn. Use it inside one chain step that needs open-ended tool use, or instead of the chain when the steps cannot be known in advance.
- Plan-and-Execute: often confused with chaining. There an LLM writes the step list at run time and the steps then run, often as a chain. Here the developer writes the list.
- Orchestrator–Workers: a lead model decides the subtasks per request and hands them to workers. In a chain the steps are fixed at design time.
- Supervisor: a central agent picks which worker acts after every step. A chain's next step is fixed in code; "sequential orchestration" in agent frameworks is the chain form of the same agents.
- Hierarchical teams: a chain can be one team inside a hierarchy. Many role-based "company" systems are mostly chains with job titles.
- Swarm / Handoffs: the active agent decides who goes next, the opposite of a fixed order. Microsoft points to handoffs when a sequential pipeline needs to go back to an earlier agent or ask the user.
- Group Chat: agents share one transcript and a selector picks the next speaker. ChatDev-style "chat chains" are pipelines whose steps are small chats.
- Blackboard: passing state between fixed steps through named keys is the lightest use of a shared store, without the blackboard's opportunistic control.
- Council / Debate: several agents answer the same question and react to each other. Use it instead of a chain when steps need to argue rather than hand off.
- Human-in-the-Loop: approval or edit checkpoints between steps. A person can replace or back up a programmatic gate.
Further reading
- Building effective agents (Anthropic, 2024): the definition of prompt chaining and gates, workflows versus agents, and the advice to start simple.
- basic_workflows.ipynb (Anthropic cookbook): the minimal
chain()loop and a four-step data-processing example. - AI agent orchestration patterns (Azure Architecture Center): sequential orchestration with when to avoid it, context management, reliability and cost guidance.
- AI Chains: Transparent and Controllable Human-AI Interaction by Chaining LLM Prompts (Wu, Terry, Cai, CHI 2022): a taxonomy of primitive steps and a user study, including cascading errors.
- Prompt Chaining or Stepwise Prompt? Refinement in Text Summarization (Sun et al., ACL Findings 2024): separate calls against one combined prompt, and the simulated refinement hypothesis.
- Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering (Ridnik, Kredo, Friedman, 2024): a multi-stage code flow with YAML output and test anchors.