Prompt Chaining

Also known as: pipeline, sequential workflow, chain, sequential orchestration (Microsoft), linear delegation (Azure Architecture Center), deterministic flow (OpenAI Agents SDK), SequentialAgent or sequential process (Google ADK, CrewAI)

Intent

Split one hard task into a fixed sequence of smaller LLM calls and plain code steps. Each step gets a narrow instruction and the output of the step before it, and code can check the result between steps before anything moves on.

The order of the steps is written by the developer. The model never decides what runs next.

Problem

A company wants a French risk memo for every English supplier contract it receives. The legal team reads French; the contracts run to about 40 pages.

The first version is one prompt: "read this contract, find the risky clauses, rate them, write a memo in French following our template". It works on some contracts. On others it breaks in ways that are hard to see:

A 40-page contract goes into one prompt that must extract, rate, write, translate and format at once; the memo that comes out misses clauses, invents clause ids and is sometimes in English. Contract 40 pages, English One prompt does it all extract clauses, rate risk, write the memo, translate it, fill the template French memo? missed clauses invented clause ids sometimes English
Everything happens inside one call, so there is no intermediate result to check and no way to tell which part went wrong.

Solution

Cut the task where the type of work changes, give each piece its own short prompt, and run the pieces in an order fixed in code. Between steps, code parses the output and checks it. A check that fails can retry the step, stop the chain, or hand the case to a person. Anthropic's "Building effective agents" calls these checks gates and presents prompt chaining as the first and simplest workflow.

For the contract memo, the chain has five steps:

  1. Extract. A small model lists the clauses as JSON: [{id, title, text}]. Gate: the JSON parses and holds at least one clause.
  2. Classify. A strong model rates each clause low, medium or high against a rubric. Gate: every clause id from step 1 is present and every rating is one of the three values.
  3. Draft. The memo is written in English from the high and medium items only.
  4. Translate. A cheap model translates it into French. Gate: a language detector says French.
  5. Format. Plain code puts the text into the legal template. No LLM call.

Each step is small enough to prompt well, test on its own and debug. When a memo is wrong, the stored intermediate results show which step failed.

Who decides the flow: the developer, at design time. The model writes the content of each step, and at most produces a value that a gate reads. It never picks the next step. That is what makes this a workflow and not an agent. In Plan-and-Execute an LLM writes the step list at run time; in Orchestrator–Workers a lead model invents the subtasks per request. Here the list is in your code.

Real-world analogy

A newspaper desk. The reporter files a story, the fact-checker checks names and figures, the copy editor fixes style, the translator produces the foreign edition, and layout puts it on the page. The desk order is set by the managing editor, not by the people at each desk. The fact-checker is a gate: a story with a wrong figure goes back to the reporter instead of moving on.

The analogy stops at judgement. A human copy editor who notices that a paragraph is missing will say so. An LLM step usually works with whatever it is handed and carries on, which is why a chain needs explicit gates and why each step should see the original source when it needs ground truth.

Structure

Input flows through Step 1, Gate 1, Step 2, Gate 2 and a code step to the output. Each gate can send its step back for a retry with the error message, or, after the retry limit, send the run to an early exit. A shared state store holds every intermediate result. retry with error (max k) retry with error (max k) fail fail Input Step 1 LLM: extract Gate 1 check JSON Step 2 LLM: rate risk Gate 2 check ids Step 3 code: format Early exit error, fallback answer or human review Output Shared state {input, clauses, risks, ...}: written after each step, plus one trace span per step 1 2 3 4 5 6
Solid lines are the fixed order written in code; the accent arrows leave a gate when its check passes. Dashed lines are retries that feed the gate's error back to the same step. Red lines are failures after the retry limit. The chain shown is simplified, so its step numbers differ from the five-step example.
  1. Step. One focused transformation: a prompt template, a model and its settings, and a parser that turns raw text into a typed value. A step can also be plain code or a whole tool-using agent.
  2. Gate. Code, or a cheap classifier call read by code, that checks the parsed value: schema, counts, a regex, a business rule. It returns pass, retry, repair, early exit or escalate.
  3. Handoff contract. Exactly what the next step receives: the previous output, and the original input or earlier results only when needed. Too little loses information; too much bloats the prompt.
  4. Code step. Deterministic work such as formatting, a database lookup or running tests. Cheaper and easier to test than an LLM call.
  5. Early exit. A controlled stop when a gate keeps failing or the input is out of scope: an error, a fallback answer, or a person.
  6. State and trace. Intermediate results stored by name, so runs can be inspected, cached and resumed, with one trace span per step (prompt, output, tokens, latency, gate verdict). The runner that calls the steps in order is often just a for loop.

Prompt generator

Describe the goal and list the steps in order. For each step give its single job, its output format and the check that runs after it. Set the model to code for steps that need no LLM. The result has two parts: rules for the runner code (retries, early exit, budget, tracing), then one prompt per LLM step with its input tags. Paste the step prompts into your own runner, a LangGraph node, or any framework's sequential steps.

Steps in run order, one row per step
When a gate still fails

Generated prompt

Highlighted text comes from the form. [Bracketed] text is still to fill in.

The gates and contracts matter more than the prompts

Most chain failures happen at handoffs: a malformed JSON, a dropped clause, a field the next step needed. Put a cheap code check right after extraction and planning steps, because errors there spread to everything downstream. Keep a single-prompt baseline and keep only the steps that beat it on your eval set.

Applicability

Use it when

Avoid it when

The boundary moves both ways. If you keep adding branches for unforeseen cases, or gates fail often because inputs vary more than the steps assume, one step (or the whole chain) should become an agent. A common middle ground keeps the outer chain and makes one step an agent. If traces show an agent takes the same path almost every time, fix that path in code: OpenAI's Agents SDK docs say orchestrating with code is more predictable in speed, cost and performance.

How to implement

  1. Write the single-prompt baseline first. The chain must beat it on an eval set to justify its cost.
  2. Find the seams. Good split points are where the output type changes (text to data to prose), where a different skill is needed (extraction, reasoning, style, translation), where correctness is cheap to check, or where a different model fits better.
  3. Define a data contract for each handoff. Use structured output (JSON schema, Pydantic) for anything code reads. Use free text only when the next consumer is another LLM, and then wrap each upstream result in its own XML tag (<outline>, <draft>) so the next prompt can refer to it by name.
  4. Write one narrow prompt per step. State its single goal, input format and output format, and what not to do ("do not translate yet").
  5. Add gates where errors are cheap to detect and costly to pass on. Right after extraction and planning steps. On failure, retry with the error message fed back.
  6. Set retry and stop rules. A maximum number of retries per step, a total token or cost budget, and a timeout per call and per chain. Keep transport retries (rate limits) separate from retries for bad output.
  7. Choose a model per step. Small models for classification, extraction and formatting, strong ones for reasoning and synthesis (Azure Architecture Center cost guidance).
  8. Trace and test each step on its own, then end to end. Persist state after each step so a crash resumes instead of restarting.
  9. Cache deterministic or repeated steps by step name, prompt version, model and input hash.

A framework is optional. Anthropic's cookbook implements the whole pattern as a loop that appends the previous result to the next prompt (its example turns a messy performance report into a markdown table in four steps), and Anthropic warns that framework abstractions can hide prompts and responses. Agent frameworks package the same idea as "sequential orchestration of agents"; if a step needs no tools or multiple turns, a plain LLM call is cheaper and easier to test.

Framework support (check current docs before use)
FrameworkHow this pattern appears
Raw APIA for loop over prompts; gates are your own if statements.
LangChain (LCEL)prompt | model | parser builds a RunnableSequence; .with_retry(), .with_fallbacks(), with_structured_output. Streams end to end only if every component supports it.
LangGraphStateGraph with add_edge for the line and add_conditional_edges for a gate; typed shared state.
OpenAI Agents SDKSeveral Runner.run calls in code, a checker agent with output_type, an if for the gate, all inside trace().
Google ADKSequentialAgent(sub_agents=[...]) with output_key and {key} templating. Now marked deprecated in favour of the ADK 2.0 graph Workflow.
CrewAIProcess.sequential; context=[...] picks earlier tasks; guardrail with guardrail_max_retries (default 3).
Semantic KernelSequentialOrchestration(members=[...]); labelled experimental, not in the Java SDK.
Microsoft Agent FrameworkSequentialBuilder; passes the full conversation by default, chain_only_agent_responses=True passes only the previous response; cannot go back to an earlier agent.
DSPyA module whose forward() calls sub-modules in order; a compiler tunes every step's prompts against a metric.
LlamaIndex Workflows@step methods linked by typed events, from StartEvent to StopEvent.

Caveat. These APIs move fast. Google ADK's docs mark SequentialAgent deprecated without a version or removal date, Semantic Kernel's agent orchestration is experimental and Microsoft Agent Framework appears to be its successor, and DSPy's assertion API has changed between versions (current names not verified). Graph frameworks are the usual upgrade path: a chain is a linear graph, a conditional edge turns it into routing, and a back edge turns it into a loop.

Variants

Linear chain

A, then B, then C, with no checks. The cookbook chain() loop or an LCEL pipe. Fine for prototypes and low-stakes transforms.

Gated chain

Code checks between steps decide pass, retry or early exit. OpenAI's deterministic-flow example simply stops when the outline checker says the outline is poor or off-genre. A gate that stops the run with a clear reason is doing its job.

Hybrid chain (LLM and code steps)

Some steps are parsing, retrieval, database lookups, running tests or formatting. Decomposed Prompting (Khot et al., 2023) lets a sub-task be a prompt, a trained model or a symbolic function. The Format step in the contract example is one.

Extract, transform, generate

Unstructured input becomes structured data, then processed data, then prose. Document pipelines take this shape. DocETL rewrites such pipelines into finer chains and reports 25 to 80% more accurate outputs than well-engineered baselines on four tasks (Shankar et al., 2024, self-reported).

Draft, critique, refine

A fixed three-step chain: generate, review against criteria, rewrite once. Claude's prompting docs name it the most common chaining pattern. If the refine step loops until a criterion passes, it becomes Evaluator–Optimizer.

Chain with embedded loops ("flow engineering")

A mostly linear pipeline where some stages iterate against tests. AlphaCodium runs a linear pre-processing chain (reflect on the problem, reason about the tests, propose and rank solutions, generate extra tests), then fixes code against tests. The fix loop itself is an evaluator–optimizer loop.

Per-item chain

The same chain runs on each document or clause, often in parallel, then a reduce step merges the results. The fan-out belongs to Parallelization; each lane is a chain.

Chain with human checkpoints

A pause after a step for approval or edits, for example before the memo is sent. Microsoft Agent Framework has with_request_info; LangGraph has interrupts. See Human-in-the-Loop.

Chain of agents

Each step is a full agent with tools and its own loop, and the order between agents stays fixed. This is what Semantic Kernel, ADK and CrewAI call sequential orchestration. Inside each step, see ReAct.

Compiled chain

The prompts and few-shot examples of every step are tuned automatically against a metric. DSPy does this with a compiler over modules composed in forward().

Generated chain

An LLM writes the step list once, then code runs it as a fixed chain. That is Plan-and-Execute.

Pros and cons

Pros

  • Usually more accurate, because each call is an easier task. AlphaCodium (a multi-stage flow, self-reported) raised GPT-4's pass@5 on its CodeContests validation set from 19% to 44%.
  • Every intermediate result can be logged, unit-tested and replayed. In the AI Chains user study (Wu et al., 2022, 20 participants), people debugged by testing individual steps and rated chaining higher for transparency and control.
  • Each step can use its own model, temperature and tools, so most steps can run on a cheap model.
  • The same path runs every time, which makes cost and latency predictable and audits possible.
  • Steps can be cached, and long shared prefixes can use prompt caching.
  • No framework needed: the runner is a few lines of code.

Cons

  • Higher latency: the sum of every step plus parsing and retries, with no parallelism inside a strict chain.
  • Usually more tokens overall: repeated instructions, repeated context, retries. AlphaCodium uses about 15 to 20 LLM calls per solution.
  • Errors compound unless gates catch them. As arithmetic, not a measurement: if each of 5 independent steps succeeds 95% of the time, the chain succeeds about 77% of the time; with 10 steps, about 60%.
  • Rigid: inputs that do not fit the assumed steps produce garbage.
  • Steps are coupled: changing one step's output format can break the next step's parser.

The compounding arithmetic assumes independent failures, and real failures are correlated. It also cuts both ways: each smaller step is easier, so its success rate is higher than the single prompt's, and a gate with retries raises it further. The goal is steps that are easy and checkable, not the fewest steps.

Most of the evidence comes from models released before built-in extended thinking (LaMDA in AI Chains, GPT-4 in AlphaCodium and Sun et al.). The sources include no recent controlled comparison of a chain against a single call to a reasoning model, so treat the size of the gains as model-dependent and measure on your own task.

Failure modes

Cascading errors

A wrong extraction early on poisons every later step. In the contract chain, a clause missed in step 1 never reaches the memo, and the memo still reads well.

Mitigation: gates right after early steps; checks against known-good data (AlphaCodium's "test anchors" stop a wrong generated test from undoing a fix that passed known tests); let a person fix intermediate outputs.

Information loss

A later step needs a detail that an earlier step dropped or summarized away, like a game of telephone. The draft step knows the risk rating but not the exact clause wording.

Mitigation: pass the original input or the earlier result along when a step needs ground truth; store large artifacts and pass references; test with inputs that contain rare details.

Context bloat

Passing the full history to every step makes prompts long, slow, costly and distracting.

Mitigation: minimal handoff contracts; summarize or prune between steps. Sources disagree here: Cognition argues that when steps are long-running agents making implicit choices, narrow handoffs lose those choices. For short, well-specified LLM calls, minimal contracts work well.

Format drift and parse failures

The model wraps the JSON in prose or drops a field, and the next step crashes or, worse, carries on with half the data.

Mitigation: native structured output, schema validation with the error fed back on retry (as Instructor does), strict parsers that fail loudly, YAML instead of JSON when the output contains code (AlphaCodium's choice: no quote escaping, fewer tokens).

Miscalibrated gates

Too strict and you get endless retries and false rejections. Too loose and errors slip through. An LLM-judge gate has its own error rate.

Mitigation: prefer code checks; measure each gate's false-pass and false-reject rates on labelled examples; cap retries; log every verdict and review rejected cases.

Over-decomposition

Many tiny steps add latency and cost, and every handoff is a chance to lose information.

Mitigation: merge steps that never fail on their own. Keep a step only if it improves an eval metric or makes a check possible.

Simulated process

One prompt asked to "draft, then critique, then refine" goes through the motions. Sun et al. (2024) found three separate prompts beat one combined prompt for summarization, though the combined prompt wrote better critiques, so separation seems to matter most for the generation steps.

Mitigation: make each phase a separate call.

Inputs that do not fit the steps

A document with no clauses section, or a contract in German, goes through the chain anyway and produces nonsense.

Mitigation: an input classifier gate at the start, with early exit or Routing to another chain.

Silent regressions

Editing one prompt changes its output distribution and quietly breaks a later step.

Mitigation: version prompts and contracts; per-step assertion tests on real samples, rerun on every pipeline change; an end-to-end eval on every change.

Retry storms

Retries on every step multiply calls, and transport retries stack on top of validation retries.

Mitigation: per-step and global budgets for tokens, time and money; one retry layer for transport errors, not two (turn off the SDK's retries if a wrapper already retries).

Hidden errors

A step returns an empty or placeholder value and later steps carry on.

Mitigation: surface errors instead of hiding them; make "unknown" an explicit field in contracts; gates that reject empty values.

Injected instructions travelling down the chain

Text in the contract (or a tool result) contains instructions, and a later step obeys them.

Mitigation: wrap upstream outputs in tags and tell each step they are data; apply content checks at several points, not only at the end; give each step the fewest tools it needs.

Interrupted long chains

A crash or deploy mid-chain loses all the work done so far.

Mitigation: persist state after each step (LangGraph or Agent Framework checkpointers), make steps idempotent, resume from the last completed step.

Relations with other patterns

Further reading