Parallelization
Also known as: fan-out / fan-in, scatter-gather, concurrent orchestration (Microsoft), parallel agent (Google ADK), sectioning, voting, self-consistency, best-of-N, map-reduce
Intent
Run several LLM calls at the same time on pieces of work that do not depend on each other, then combine their outputs in one aggregation step. The branches are either different subtasks (sectioning) or the same task run several times (voting). Your code decides the split before the run starts, which is what separates this pattern from an orchestrator that invents subtasks on the fly.
Problem
You are building a due-diligence tool for a small investment team. An analyst uploads the data room of Acme Robotics (corporate documents, three years of financial statements, the main customer contracts, a market study) and asks: is Acme a reasonable acquisition target?
The tool needs a legal review, a financial review and a market review. None of them needs the others' output. The first versions of the tool break in three ways:
- One prompt does all three jobs. The model has to read contracts for change-of-control clauses, check revenue figures and judge competitors in the same pass. Each concern gets less attention. Anthropic reports that one call per concern tends to work better than one call juggling several (a heuristic from experience, not a published number).
- Three calls run one after the other. Each review is a separate prompt, but the code awaits them in sequence. If each takes about 20 seconds, the analyst waits a minute for work that could finish in about 20 seconds plus a short merge.
- One sample is trusted blindly. When a single call misreads a clause, nothing catches it, and you have no signal of how confident the answer is.
Solution
Write the split in code. Start one LLM call per independent piece at the same time, wait for them at a barrier, then hand all results to one aggregator. Each branch sees only its own prompt and the shared input; branches never talk to each other while they run.
- Split. Code builds the list of branch tasks:
legal_review,financial_review,market_review, each with its own focused prompt and the same data room. - Fan out. All three calls start together, under a concurrency cap and a per-branch timeout.
- Branch work. Each reviewer returns JSON in a known schema: a list of findings with severity and evidence.
- Fan in. A barrier waits until every branch is done, a quorum is reached, or a deadline passes. Failed branches are recorded, not hidden.
- Aggregate. An LLM merges the three reports into one risk summary. It keeps every distinct risk, names which reviewer raised it, states disagreements, and marks a missing review as "not available".
- Return the report with metadata: which branches succeeded, and whether the result is degraded.
That is sectioning: different subtasks in each branch. The second flavor, voting, runs the same task several times and aggregates the answers. In the same tool, three reviewers could each read the customer contracts for change-of-control clauses, and the code flags a clause if any of them reports it. Both names come from Anthropic's "Building effective agents".
Who decides the flow: the developer. Code fixes how to split, how many branches, which prompts and models, the limits and the aggregation rule. The LLMs only decide the content of each branch, and optionally the merged text. In Orchestrator–Workers the fan-out looks the same, but a lead LLM decides at run time how many subtasks exist and what each one covers.
Real-world analogy
A buyer about to purchase a building sends the same file to a lawyer, an accountant and a surveyor on Monday morning. They work in separate offices and never call each other. On Friday the buyer reads the three reports side by side and writes one decision memo. The buyer chose the three experts before sending anything: that is the developer's split. The memo is the aggregator.
For voting, picture getting two surveyors' opinions on the roof and acting if either one finds damage. The analogy stops working on cost: human experts are slow and expensive, while an extra LLM branch is cheap per call but multiplies your token bill and your rate-limit pressure.
Structure
- Splitter. Code that turns the input into a list of branch tasks and guarantees they are independent. For map-reduce it is a text splitter that makes one task per chunk.
- Branches. One LLM call (or a full agent with tools) per subtask or per sample. Each gets its own prompt and returns a result in a known schema. A scheduler starts them together and enforces the concurrency cap, timeouts and retries.
- Fan-in barrier. Waits for all branches, a quorum, a deadline, or an early exit such as a guardrail tripping. It records which branches failed.
- Aggregator. Turns N outputs into one: a pure function (concatenate, union, vote, k-of-N threshold) or an LLM (synthesis, selection). Pick the rule before writing the prompts, because it decides the output schema.
- Result with metadata. The merged answer plus
branches_ok,branches_failed,degraded, and for voting the vote counts and margin.
Prompt generator
Describe the shared input and one row per branch. The output has four parts: a run spec for your fan-out code (limits, failure policy, aggregation rule), the shared input block, one prompt per branch, and the aggregator prompt or rule. Use them with plain asyncio.gather, a LangGraph fan-out, or any framework's parallel step.
Generated prompt
Highlighted text comes from the form. [Bracketed] text is still to fill in.
Most of the work is in the contracts, not the wording. Give branches focus areas that do not overlap, make every branch return typed JSON, and decide the aggregation rule first. A synthesis prompt fed N free-form essays is where details get lost; a vote over a normalized final_answer field is easy to get right.
Applicability
Use it when
- Subtasks are truly independent. No branch needs another's output and order does not matter. Microsoft's guide calls this "embarrassingly parallel" work.
- Latency matters and branches take similar time. Wall-clock time becomes the slowest branch plus aggregation, instead of the sum.
- One prompt is juggling several concerns. A guardrail check beside the main answer, or one grader per evaluation aspect, each get the model's full attention.
- You need reliability or a confidence signal. Voting works when answers have a comparable form (labels, numbers, options, code with tests), and the vote margin tells you how sure to be.
- You want several "eyes" with a tunable threshold. Security review or moderation, where you trade false positives against false negatives by choosing k.
- The input exceeds the context window and each chunk can be processed without the others (map-reduce).
Avoid it when
- Steps depend on each other. Use Prompt Chaining for that edge.
- You cannot list the subtasks in advance. Let a model decide with Orchestrator–Workers or Plan-and-Execute.
- Branches would write to the same state or external system. Without coordination they overwrite each other. Keep writes in the aggregator or after it.
- You have no rule for contradictory results. If you cannot say how to merge, the merge will be arbitrary.
- Rate limits or budget are tight. Voting multiplies cost roughly linearly with N, and a single good call may already meet the bar.
- Errors are correlated. If every sample shares the same misconception, more votes add cost but little accuracy.
How to implement
- Prove independence. Draw the dependency graph of the subtasks. Any edge between two of them is a chain, not a branch.
- Pick the flavor. Different concerns: sectioning. Same question, need reliability: voting. Input too large or many items: map-reduce.
- Define a structured output per branch. Include a stable
branch_id, astatus(ok,error,timeout,invalid), the payload, and for voting a normalizedfinal_answer. - Choose the aggregation rule before the prompts. Concatenate, union and dedupe, majority vote, k-of-N, best-of-N with a verifier, or LLM synthesis. The rule decides the schema.
- Set limits. A concurrency cap below your provider's rate limit, a per-branch timeout, an overall deadline, retries with backoff on HTTP 429, and a failure policy: fail-all, best-effort with a
degradedflag, or quorum. - Sort before merging. Completion order is not stable (LangGraph and Semantic Kernel both say so), so sort results by
branch_idand use a fixed tie-break rule. - Trace under one parent span with a child span per branch, so you can see the N calls side by side. OpenAI's parallelization example wraps all runs in a single trace.
- Measure against a single-call baseline on accuracy, p50/p95 latency and cost per task before keeping the extra branches. Remove one branch at a time to check that each earns its cost.
In plain Python the core is an asyncio.gather over branch calls wrapped in a semaphore and asyncio.wait_for. Most frameworks offer a named primitive for the same thing:
| Framework | Fan-out | Fan-in |
|---|---|---|
| LangGraph | Several edges from one node; Send for a dynamic map | Reducers on state keys; list-form edge add_edge(["a","b"], "merge"); defer=True |
| LangChain (LCEL) | RunnableParallel; batch with max_concurrency | A dict keyed by branch name |
| OpenAI Agents SDK | asyncio.gather over Runner.run | Your code, or a picker agent |
| Google ADK | ParallelAgent(sub_agents=[...]) | Distinct output_key per branch in session state, then a merge agent |
| CrewAI | Task(async_execution=True) | A later task with context=[...] |
| LlamaIndex Workflows | send_event several times; num_workers | collect_events |
| Microsoft Agent Framework | ConcurrentBuilder, AddFanOutEdge | Default or custom aggregator; AddFanInBarrierEdge |
| Semantic Kernel | ConcurrentOrchestration | List of results, order not guaranteed |
Two LangGraph behaviours are worth knowing. A superstep is transactional: if one parallel branch raises, none of that step's state updates are applied, and a RetryPolicy re-runs only the failing branch. A list-form edge silently never fires if one of its listed nodes never runs.
Caveat. These APIs move quickly. Google ADK marks template workflow agents, ParallelAgent included, as deprecated in favour of graph-based workflows, with no removal date given. Semantic Kernel's orchestration API is experimental. Verify names against current docs.
Variants
Sectioning
A fixed list of different calls, as in the running example. Anthropic's examples are a guardrail running beside the main answer and automated evals where each call grades one aspect (faithfulness, tone, completeness).
Parallel guardrails
A sectioning case where one branch can veto the other. The OpenAI Agents SDK runs input guardrails concurrently with the agent by default and offers a blocking mode. The trade-off it documents: in parallel mode the agent may already have spent tokens and run tools before the guardrail trips. Never run side-effecting tools in a branch a parallel guardrail can still cancel.
Voting and self-consistency
The same prompt, N samples at a temperature above zero, majority vote on the extracted answer. Wang et al. sampled 40 reasoning paths and reported +17.9 points on GSM8K over plain chain-of-thought, but most of the gain arrived with 5 to 10 paths. In their aggregation comparison a plain majority vote (74.4%) matched the best weighted variant, while weighting by unnormalized model probabilities fell to 59.9%. These results come from 2022 models; gains on current models are likely smaller on saturated benchmarks. The method needs answers from a fixed set.
Prompt and input perturbation
Vary the prompt instead of, or as well as, the temperature. Medprompt's choice-shuffle ensemble runs 5 calls with the multiple-choice options reordered and takes the most frequent answer; it added about 2.2 points in the authors' ablation. Shuffling also counters position bias.
Multi-model panels
Different models answer or judge, then vote. In the PoLL study, a panel of smaller judges from different model families beat a single large judge at over seven times lower cost, with less self-preference bias. Once judges read and critique each other, it becomes Council / Debate.
Universal self-consistency
For free-form outputs where exact-match voting is impossible, an LLM reads all candidates and picks the one most consistent with the others. The paper reports it matches standard self-consistency on math and execution-based voting on code.
Best-of-N with a verifier
Generate N candidates and keep the one a checker scores highest: unit tests, a trained verifier or an LLM judge. With an automatic checker this scales far: in "Large Language Monkeys", SWE-bench Lite solves rose from 15.9% with one sample to 56% with 250. Without a reliable checker, selection plateaus after a few hundred samples.
Map-reduce
The number of branches comes from the data (chunks, files, rows), but the per-branch operation is fixed. When the mapped summaries are still too long for one reduce call, a collapse step reduces them in groups. Bound that loop: LangChain's example uses a recursion limit of 10.
Skeleton-of-Thought
The model first writes a short outline, then each point is expanded in parallel. The paper reports speed-ups up to 2.39x, with quality holding on knowledge and common-sense questions but dropping on math, coding and writing. The outline is generated at run time, so this edges toward orchestrator–workers.
Parallel tool calls
The model emits several tool calls in one turn and your code runs them concurrently. Here the model decides the fan-out, so this sits between this pattern and the ReAct loop. Claude's docs require all tool results in one user message; splitting them across messages teaches the model to stop calling tools in parallel.
Pros and cons
Pros
- Wall-clock time drops to the slowest branch plus the merge when subtasks are independent.
- Each branch has a short, focused prompt, and simple branches can use a smaller model.
- Voting raises accuracy on answers with a fixed form and gives a confidence signal (the vote margin) for free.
- The flow is fixed in code, so it is predictable, easy to test branch by branch, and cheap compared with agent patterns.
- A k-of-N threshold lets you tune recall against precision without touching the prompts.
Cons
- Roughly N times the tokens of one call, plus aggregation. Voting does not reduce latency; it adds the slowest sample and the aggregator.
- N requests at once hit rate limits sooner.
- More calls mean more chances of a failure per run, so you need timeouts, retries and a failure policy.
- The aggregation step is a design problem of its own, and an LLM synthesizer can lose details or invent agreement.
- Unordered completion and sampling make runs hard to reproduce unless you sort and fix tie-breaks.
Failure modes
- Hidden dependency
-
The financial review needed a fact the legal review found (say, a contract that ends on a change of control). The two reports contradict each other and the merge cannot tell why.
Mitigation: draw the dependency graph before building; chain dependent steps instead of parallelizing them.
- Rate-limit storm
-
All branches fire at once, many get HTTP 429, and naive retries make the burst worse.
Mitigation: a semaphore below the provider's limit, exponential backoff with jitter, or the framework's
max_concurrency/num_workers. - Straggler
-
One slow branch sets the total latency. Anthropic notes that in its research system the lead waits for subagents synchronously, so one slow subagent blocks the whole step.
Mitigation: a per-branch timeout, an overall deadline, or a quorum, and return a result marked as degraded.
- Silent partial failure
-
One branch errors and the aggregator merges the rest as if nothing happened. The report has no market section, or a vote is biased by the missing samples.
Mitigation: an explicit policy (fail-all, best-effort with a
degradedflag, or quorum) and an aggregator instruction to write "not available" for missing parts. - Side effects before a guardrail trips
-
The main branch has already spent tokens or called a tool by the time the parallel guardrail blocks the request.
Mitigation: keep side-effecting tools out of the main branch until the guard returns, or use blocking mode.
- Shared state conflicts
-
Two branches write the same key or record and one overwrites the other. Microsoft lists shared mutable state between concurrent agents as an anti-pattern.
Mitigation: a distinct output key per branch, reducers that merge concurrent writes, and writes only in the aggregator.
- Answer extraction errors
-
"42", "42.0" and "forty-two" count as three different votes, and unparseable outputs slip into the count.
Mitigation: ask for a normalized
final_answerfield, normalize before counting, and count unparseable outputs separately. - Correlated errors
-
Every sample shares the same bias, so the majority is confidently wrong.
Mitigation: diversify the branches (different prompts, shuffled options, different model families) and use an executable verifier where you can.
- Too many votes
-
Chen et al. show accuracy can rise and then fall as calls increase on tasks of mixed difficulty, because extra calls help easy queries and hurt hard ones.
Mitigation: sweep N on a dev set (for example 1, 3, 5, 10) and plot accuracy against cost instead of maximizing N.
- Aggregator overload
-
The synthesis call receives N long reports, runs out of context or drops details.
Mitigation: structured, length-capped branch outputs, and a hierarchical reduce when there are many branches.
Relations with other patterns
- Prompt Chaining: the sequential counterpart, for steps that depend on each other. The two combine often: pre-process in a chain, fan out, merge.
- Routing: a classifier picks one branch per input; parallelization runs several. Multi-label routing sits in between, choosing which branches to fan out to.
- Evaluator–Optimizer: refines one answer in sequence. Best-of-N is its parallel cousin, and voting at matched cost is the baseline a refinement loop should beat.
- ReAct (tool-use loop): parallel tool calls are a local fan-out inside that loop, decided by the model instead of the developer.
- Plan-and-Execute: DAG planners such as LLMCompiler run independent planned steps in parallel; the plan, not your code, defines the branches.
- Orchestrator–Workers: same fan-out and fan-in shape, but a lead LLM decides the subtasks at run time. LangGraph's
Sendserves both; the difference is whether code or an LLM wrote the list. - Supervisor: when a supervisor calls several workers in one hop, that hop is a small parallel fan-out with the supervisor as aggregator.
- Hierarchical teams: a team lead commonly runs its workers in parallel.
- Council / Debate: members see and respond to each other before a decision. Voting keeps samples independent; round 0 of a council is a parallel fan-out, and cost-matched voting is its baseline.
- Blackboard: branches writing distinct keys to shared state is a light blackboard; a full one has agents reacting to each other's writes.
- Human-in-the-Loop: a low vote margin or conflicting sections are natural points to ask a person.
Further reading
- Building effective agents (Anthropic): defines parallelization, sectioning and voting, with the guardrail, eval and code-review examples.
- AI agent orchestration patterns (Microsoft Azure Architecture Center): concurrent orchestration, when to avoid it, aggregation strategies and reliability advice.
- Self-Consistency Improves Chain of Thought Reasoning in Language Models (Wang et al.): the voting method and its aggregation comparison.
- Are More LLM Calls All You Need? (Chen et al.): why more votes can lower accuracy, and how to pick the number of calls.
- Guardrails (OpenAI Agents SDK): parallel versus blocking guardrails and the side-effect trade-off.
- Graph API control flow (LangGraph): supersteps, reducers, deferred nodes and transactional errors in parallel branches.