Parallelization

Also known as: fan-out / fan-in, scatter-gather, concurrent orchestration (Microsoft), parallel agent (Google ADK), sectioning, voting, self-consistency, best-of-N, map-reduce

Intent

Run several LLM calls at the same time on pieces of work that do not depend on each other, then combine their outputs in one aggregation step. The branches are either different subtasks (sectioning) or the same task run several times (voting). Your code decides the split before the run starts, which is what separates this pattern from an orchestrator that invents subtasks on the fly.

Problem

You are building a due-diligence tool for a small investment team. An analyst uploads the data room of Acme Robotics (corporate documents, three years of financial statements, the main customer contracts, a market study) and asks: is Acme a reasonable acquisition target?

The tool needs a legal review, a financial review and a market review. None of them needs the others' output. The first versions of the tool break in three ways:

The data room goes through a legal, a finance and a market review one after the other, so the report only arrives after about 60 seconds although no review depends on another. Data room Legal review, 20 s Finance review, 20 s Market review, 20 s Report after 60 s 60 s of waiting, though no review needs another's output (merging the three into one prompt is faster, but each concern gets less attention)
The three reviews are independent, yet the sequential version makes each one wait for the previous one.

Solution

Write the split in code. Start one LLM call per independent piece at the same time, wait for them at a barrier, then hand all results to one aggregator. Each branch sees only its own prompt and the shared input; branches never talk to each other while they run.

  1. Split. Code builds the list of branch tasks: legal_review, financial_review, market_review, each with its own focused prompt and the same data room.
  2. Fan out. All three calls start together, under a concurrency cap and a per-branch timeout.
  3. Branch work. Each reviewer returns JSON in a known schema: a list of findings with severity and evidence.
  4. Fan in. A barrier waits until every branch is done, a quorum is reached, or a deadline passes. Failed branches are recorded, not hidden.
  5. Aggregate. An LLM merges the three reports into one risk summary. It keeps every distinct risk, names which reviewer raised it, states disagreements, and marks a missing review as "not available".
  6. Return the report with metadata: which branches succeeded, and whether the result is degraded.

That is sectioning: different subtasks in each branch. The second flavor, voting, runs the same task several times and aggregates the answers. In the same tool, three reviewers could each read the customer contracts for change-of-control clauses, and the code flags a clause if any of them reports it. Both names come from Anthropic's "Building effective agents".

Who decides the flow: the developer. Code fixes how to split, how many branches, which prompts and models, the limits and the aggregation rule. The LLMs only decide the content of each branch, and optionally the merged text. In Orchestrator–Workers the fan-out looks the same, but a lead LLM decides at run time how many subtasks exist and what each one covers.

Real-world analogy

A buyer about to purchase a building sends the same file to a lawyer, an accountant and a surveyor on Monday morning. They work in separate offices and never call each other. On Friday the buyer reads the three reports side by side and writes one decision memo. The buyer chose the three experts before sending anything: that is the developer's split. The memo is the aggregator.

For voting, picture getting two surveyors' opinions on the roof and acting if either one finds damage. The analogy stops working on cost: human experts are slow and expensive, while an extra LLM branch is cheap per call but multiplies your token bill and your rate-limit pressure.

Structure

The data room goes to a splitter in code, which starts three isolated review calls at the same time; their results go to a fan-in barrier, then to an aggregator that produces the report with branch status. run at the same time: at most 3 in flight, 60 s timeout each Data room Splitter code, fixed Legal review isolated LLM call Finance review isolated LLM call Market review isolated LLM call Fan-in waits for all Aggregator LLM synthesis Report with branch status 1 2 3 4 5
Accent arrows are the fan-out, which code starts. Dashed arrows carry branch results back. With voting, the three boxes would hold the same prompt and the aggregator would be a vote in code.
  1. Splitter. Code that turns the input into a list of branch tasks and guarantees they are independent. For map-reduce it is a text splitter that makes one task per chunk.
  2. Branches. One LLM call (or a full agent with tools) per subtask or per sample. Each gets its own prompt and returns a result in a known schema. A scheduler starts them together and enforces the concurrency cap, timeouts and retries.
  3. Fan-in barrier. Waits for all branches, a quorum, a deadline, or an early exit such as a guardrail tripping. It records which branches failed.
  4. Aggregator. Turns N outputs into one: a pure function (concatenate, union, vote, k-of-N threshold) or an LLM (synthesis, selection). Pick the rule before writing the prompts, because it decides the output schema.
  5. Result with metadata. The merged answer plus branches_ok, branches_failed, degraded, and for voting the vote counts and margin.

Prompt generator

Describe the shared input and one row per branch. The output has four parts: a run spec for your fan-out code (limits, failure policy, aggregation rule), the shared input block, one prompt per branch, and the aggregator prompt or rule. Use them with plain asyncio.gather, a LangGraph fan-out, or any framework's parallel step.

Flavor
Branches one row per subtask (voting: per prompt variant)
When a branch fails

Generated prompt

Highlighted text comes from the form. [Bracketed] text is still to fill in.

Where the quality comes from

Most of the work is in the contracts, not the wording. Give branches focus areas that do not overlap, make every branch return typed JSON, and decide the aggregation rule first. A synthesis prompt fed N free-form essays is where details get lost; a vote over a normalized final_answer field is easy to get right.

Applicability

Use it when

Avoid it when

How to implement

  1. Prove independence. Draw the dependency graph of the subtasks. Any edge between two of them is a chain, not a branch.
  2. Pick the flavor. Different concerns: sectioning. Same question, need reliability: voting. Input too large or many items: map-reduce.
  3. Define a structured output per branch. Include a stable branch_id, a status (ok, error, timeout, invalid), the payload, and for voting a normalized final_answer.
  4. Choose the aggregation rule before the prompts. Concatenate, union and dedupe, majority vote, k-of-N, best-of-N with a verifier, or LLM synthesis. The rule decides the schema.
  5. Set limits. A concurrency cap below your provider's rate limit, a per-branch timeout, an overall deadline, retries with backoff on HTTP 429, and a failure policy: fail-all, best-effort with a degraded flag, or quorum.
  6. Sort before merging. Completion order is not stable (LangGraph and Semantic Kernel both say so), so sort results by branch_id and use a fixed tie-break rule.
  7. Trace under one parent span with a child span per branch, so you can see the N calls side by side. OpenAI's parallelization example wraps all runs in a single trace.
  8. Measure against a single-call baseline on accuracy, p50/p95 latency and cost per task before keeping the extra branches. Remove one branch at a time to check that each earns its cost.

In plain Python the core is an asyncio.gather over branch calls wrapped in a semaphore and asyncio.wait_for. Most frameworks offer a named primitive for the same thing:

Framework support (check current docs before use)
FrameworkFan-outFan-in
LangGraphSeveral edges from one node; Send for a dynamic mapReducers on state keys; list-form edge add_edge(["a","b"], "merge"); defer=True
LangChain (LCEL)RunnableParallel; batch with max_concurrencyA dict keyed by branch name
OpenAI Agents SDKasyncio.gather over Runner.runYour code, or a picker agent
Google ADKParallelAgent(sub_agents=[...])Distinct output_key per branch in session state, then a merge agent
CrewAITask(async_execution=True)A later task with context=[...]
LlamaIndex Workflowssend_event several times; num_workerscollect_events
Microsoft Agent FrameworkConcurrentBuilder, AddFanOutEdgeDefault or custom aggregator; AddFanInBarrierEdge
Semantic KernelConcurrentOrchestrationList of results, order not guaranteed

Two LangGraph behaviours are worth knowing. A superstep is transactional: if one parallel branch raises, none of that step's state updates are applied, and a RetryPolicy re-runs only the failing branch. A list-form edge silently never fires if one of its listed nodes never runs.

Caveat. These APIs move quickly. Google ADK marks template workflow agents, ParallelAgent included, as deprecated in favour of graph-based workflows, with no removal date given. Semantic Kernel's orchestration API is experimental. Verify names against current docs.

Variants

Sectioning

A fixed list of different calls, as in the running example. Anthropic's examples are a guardrail running beside the main answer and automated evals where each call grades one aspect (faithfulness, tone, completeness).

Parallel guardrails

A sectioning case where one branch can veto the other. The OpenAI Agents SDK runs input guardrails concurrently with the agent by default and offers a blocking mode. The trade-off it documents: in parallel mode the agent may already have spent tokens and run tools before the guardrail trips. Never run side-effecting tools in a branch a parallel guardrail can still cancel.

Voting and self-consistency

The same prompt, N samples at a temperature above zero, majority vote on the extracted answer. Wang et al. sampled 40 reasoning paths and reported +17.9 points on GSM8K over plain chain-of-thought, but most of the gain arrived with 5 to 10 paths. In their aggregation comparison a plain majority vote (74.4%) matched the best weighted variant, while weighting by unnormalized model probabilities fell to 59.9%. These results come from 2022 models; gains on current models are likely smaller on saturated benchmarks. The method needs answers from a fixed set.

Prompt and input perturbation

Vary the prompt instead of, or as well as, the temperature. Medprompt's choice-shuffle ensemble runs 5 calls with the multiple-choice options reordered and takes the most frequent answer; it added about 2.2 points in the authors' ablation. Shuffling also counters position bias.

Multi-model panels

Different models answer or judge, then vote. In the PoLL study, a panel of smaller judges from different model families beat a single large judge at over seven times lower cost, with less self-preference bias. Once judges read and critique each other, it becomes Council / Debate.

Universal self-consistency

For free-form outputs where exact-match voting is impossible, an LLM reads all candidates and picks the one most consistent with the others. The paper reports it matches standard self-consistency on math and execution-based voting on code.

Best-of-N with a verifier

Generate N candidates and keep the one a checker scores highest: unit tests, a trained verifier or an LLM judge. With an automatic checker this scales far: in "Large Language Monkeys", SWE-bench Lite solves rose from 15.9% with one sample to 56% with 250. Without a reliable checker, selection plateaus after a few hundred samples.

Map-reduce

The number of branches comes from the data (chunks, files, rows), but the per-branch operation is fixed. When the mapped summaries are still too long for one reduce call, a collapse step reduces them in groups. Bound that loop: LangChain's example uses a recursion limit of 10.

Skeleton-of-Thought

The model first writes a short outline, then each point is expanded in parallel. The paper reports speed-ups up to 2.39x, with quality holding on knowledge and common-sense questions but dropping on math, coding and writing. The outline is generated at run time, so this edges toward orchestrator–workers.

Parallel tool calls

The model emits several tool calls in one turn and your code runs them concurrently. Here the model decides the fan-out, so this sits between this pattern and the ReAct loop. Claude's docs require all tool results in one user message; splitting them across messages teaches the model to stop calling tools in parallel.

Pros and cons

Pros

  • Wall-clock time drops to the slowest branch plus the merge when subtasks are independent.
  • Each branch has a short, focused prompt, and simple branches can use a smaller model.
  • Voting raises accuracy on answers with a fixed form and gives a confidence signal (the vote margin) for free.
  • The flow is fixed in code, so it is predictable, easy to test branch by branch, and cheap compared with agent patterns.
  • A k-of-N threshold lets you tune recall against precision without touching the prompts.

Cons

  • Roughly N times the tokens of one call, plus aggregation. Voting does not reduce latency; it adds the slowest sample and the aggregator.
  • N requests at once hit rate limits sooner.
  • More calls mean more chances of a failure per run, so you need timeouts, retries and a failure policy.
  • The aggregation step is a design problem of its own, and an LLM synthesizer can lose details or invent agreement.
  • Unordered completion and sampling make runs hard to reproduce unless you sort and fix tie-breaks.

Failure modes

Hidden dependency

The financial review needed a fact the legal review found (say, a contract that ends on a change of control). The two reports contradict each other and the merge cannot tell why.

Mitigation: draw the dependency graph before building; chain dependent steps instead of parallelizing them.

Rate-limit storm

All branches fire at once, many get HTTP 429, and naive retries make the burst worse.

Mitigation: a semaphore below the provider's limit, exponential backoff with jitter, or the framework's max_concurrency / num_workers.

Straggler

One slow branch sets the total latency. Anthropic notes that in its research system the lead waits for subagents synchronously, so one slow subagent blocks the whole step.

Mitigation: a per-branch timeout, an overall deadline, or a quorum, and return a result marked as degraded.

Silent partial failure

One branch errors and the aggregator merges the rest as if nothing happened. The report has no market section, or a vote is biased by the missing samples.

Mitigation: an explicit policy (fail-all, best-effort with a degraded flag, or quorum) and an aggregator instruction to write "not available" for missing parts.

Side effects before a guardrail trips

The main branch has already spent tokens or called a tool by the time the parallel guardrail blocks the request.

Mitigation: keep side-effecting tools out of the main branch until the guard returns, or use blocking mode.

Shared state conflicts

Two branches write the same key or record and one overwrites the other. Microsoft lists shared mutable state between concurrent agents as an anti-pattern.

Mitigation: a distinct output key per branch, reducers that merge concurrent writes, and writes only in the aggregator.

Answer extraction errors

"42", "42.0" and "forty-two" count as three different votes, and unparseable outputs slip into the count.

Mitigation: ask for a normalized final_answer field, normalize before counting, and count unparseable outputs separately.

Correlated errors

Every sample shares the same bias, so the majority is confidently wrong.

Mitigation: diversify the branches (different prompts, shuffled options, different model families) and use an executable verifier where you can.

Too many votes

Chen et al. show accuracy can rise and then fall as calls increase on tasks of mixed difficulty, because extra calls help easy queries and hurt hard ones.

Mitigation: sweep N on a dev set (for example 1, 3, 5, 10) and plot accuracy against cost instead of maximizing N.

Aggregator overload

The synthesis call receives N long reports, runs out of context or drops details.

Mitigation: structured, length-capped branch outputs, and a hierarchical reduce when there are many branches.

Relations with other patterns

Further reading