Long-running serverless jobs: chain steps, fan out, and retry per step
Long-running serverless jobs are not one unbounded function—they are a chain of steps, each with its own timeout budget (up to 15 minutes), retry policy, and output checkpoint. When step 7 of 10 fails, you retry step 7, not the entire job. Fan out to parallel branches, merge results, pause for human approval, and resume where you left off.
Last updated: 2026-06-28
- Per-step retry: only the failed step re-runs, earlier work is checkpointed
- Fan-out + merge: parallel branches recombine in a single aggregation step
- Human-gate pauses: pipeline waits for approval before continuing
- Each step: up to 15 min timeout in an isolated container—chain for longer work
Answer first
Direct answer
Long-running serverless jobs: chain steps, fan out, and retry per step. A pipeline is a graph of steps. Each step is a serverless function invocation with its own timeout, retry policy (maxAttempts, backoffMs, fixed or exponential strategy), and output record. Completed steps are not re-run when a later step fails—the pipeline resumes from the failed step with the outputs of earlier steps available as context.
When it fits
- Work can be decomposed into independent steps where per-step retry is worth the added structure
- Fan-out: N parallel sub-tasks that recombine into a single aggregated result
- Human approval gates or external event waits mid-workflow
Tradeoffs
- A single background job function with no step decomposition means all-or-nothing retries. A transient network error in a database write at the end of a 45-minute data transform restarts the entire 45-minute transform.
- Fan-out patterns—processing N records in parallel—done in one function require N goroutines or promise chains inside one container, with no visibility into which sub-tasks succeeded before the timeout hit.
Workload and what breaks
Why long work does not fit in one function
A single serverless function, even with a 15-minute timeout, cannot reliably run multi-hour ETL, parallel data enrichment, or workflows that need human approval mid-run. One function means one failure domain: if step 7 of 10 fails at minute 50, you restart from minute zero.
Chunking work yourself—calling Lambda recursively, splitting into SQS batches—forces you to build a workflow engine in application code: tracking which chunks finished, retrying the right ones, aggregating results. That is the problem pipelines solve.
Trade-offs
Why single-step async jobs break under complexity
A single background job function with no step decomposition means all-or-nothing retries. A transient network error in a database write at the end of a 45-minute data transform restarts the entire 45-minute transform.
Fan-out patterns—processing N records in parallel—done in one function require N goroutines or promise chains inside one container, with no visibility into which sub-tasks succeeded before the timeout hit.
How Inquir helps
Pipeline steps as independent execution units
A pipeline is a graph of steps. Each step is a serverless function invocation with its own timeout, retry policy (maxAttempts, backoffMs, fixed or exponential strategy), and output record. Completed steps are not re-run when a later step fails—the pipeline resumes from the failed step with the outputs of earlier steps available as context.
Fan-out is a first-class node type: a parallel node spawns N branches concurrently; a merge node waits for all branches before continuing. Human-gate nodes pause the pipeline until an external approval event arrives. All of this is pipeline configuration—not application code.
What you get
Multi-step pipeline architecture patterns
Step chaining with per-step retry
Chain steps with dependsOn. Each step configures its own retry count and backoff (fixed or exponential). When step 7 fails, only step 7 retries—steps 1–6 stay checkpointed.
Fan-out and merge
A parallel node spawns N branches concurrently. A merge node aggregates branch outputs. Use this for N-way API enrichment, parallel image resizing, or bulk email—without managing concurrency in application code.
Human-gate pause and resume
A humanGate node pauses the pipeline. An external event (webhook, manual trigger) resumes it. Use before sensitive actions: sending emails to 100k users, charging customers, modifying production data.
Long work via step composition
Each step runs up to 15 minutes. For multi-hour ETL, decompose into steps: extract (step 1), transform batch A (step 2), transform batch B (step 3), load (step 4). Chain them; the platform handles sequencing.
What to do next
How to architect a multi-step serverless pipeline
Design the step graph first, then implement each step as an independent function.
Decompose work into steps
Map your job into a graph: which steps are sequential (dependsOn), which can run in parallel (parallel node + merge), and which need a human gate (humanGate node). Each step should fit within 15 minutes.
Implement and wire each step
Each step is a serverless function reading event.previousOutput or event.stepResults for upstream context. Configure retry count and backoff per step based on the step's failure characteristics.
Trigger from HTTP and observe
Accept the HTTP request, call global.durable.startNew(), return 202. Monitor the pipeline run in execution history: see which step is running, which retried, and what each step returned.
Code example
HTTP → extract → fan-out → merge pipeline
The HTTP handler accepts the job and returns 202. The orchestrator fans out to parallel enrichment steps, then a merge step aggregates results. Each step has its own timeout and retry policy.
export async function handler(event) { const { batchId, itemIds } = JSON.parse(event.body || '{}'); if (!batchId || !itemIds?.length) return { statusCode: 400, body: JSON.stringify({ error: 'batchId and itemIds required' }) }; const { instanceId: jobId } = await global.durable.startNew( 'enrich-batch', undefined, { batchId, itemIds } ); return { statusCode: 202, body: JSON.stringify({ jobId, status: 'started' }) }; }
export async function handler(event) { // step 1 — runs first, output feeds into fan-out branches const { batchId, itemIds } = event.payload ?? {}; const items = await db.fetchItems(itemIds); // may take several minutes await db.markBatchStarted(batchId); return { batchId, items }; // passed to parallel branches via {{steps.extract.output}} }
export async function handler(event) { // Each parallel branch receives the upstream node's output as its event. // Node retry: maxAttempts=3, backoffMs=2000, strategy=exponential const enriched = await externalApi.enrich(event.item); // retry isolates this network call return { itemId: event.item.id, enriched }; }
export async function handler(event) { // Runs after ALL parallel branches complete. A merge node hands the next step // { inputs: { <branchNodeId>: <branchOutput>, ... } } — one entry per branch. const branchOutputs = Object.values(event.inputs ?? {}); const merged = branchOutputs.map(b => b.enriched).filter(Boolean); await db.saveBatchResults(merged); return { saved: merged.length, total: branchOutputs.length }; }
When it fits
Use multi-step pipelines when…
When this works
- Work can be decomposed into independent steps where per-step retry is worth the added structure
- Fan-out: N parallel sub-tasks that recombine into a single aggregated result
- Human approval gates or external event waits mid-workflow
When to skip it
- Simple sequential work that fits in one step under 15 minutes—a single pipeline step is fine without decomposition
FAQ
FAQ
How long can a single pipeline step run?
Each step is a function invocation with a configurable timeout, up to 15 minutes (900,000ms). For work that needs more than 15 minutes, decompose it into multiple steps chained in the pipeline—each step can run up to 15 minutes independently.
How does per-step retry work exactly?
Each pipeline step has its own retry configuration: maxAttempts, backoffMs, and strategy (fixed or exponential). When a step fails and exhausts its retries, the pipeline marks that step failed and the run stops—earlier completed steps are not re-run. You can then replay the pipeline from the failed step.
Can a human gate wait indefinitely?
A humanGate node pauses the pipeline until an external event arrives via the platform's event API. There is no hard wall on how long a gate can wait, but pipeline runs in WAITING status do count toward your workspace limits. Use gate timeouts for compliance workflows with SLA requirements.
How do fan-out branches receive their input?
The parallel node passes the upstream node's output to each outgoing branch, so a branch handler reads it directly as its event. After every branch finishes, the merge node hands the next step an inputs object keyed by branch node id—iterate Object.values(event.inputs) to read the branch outputs.