Long-running serverless jobs without HTTP timeouts
HTTP serverless functions time out in seconds. Move work that takes minutes or hours—PDF generation, bulk data sync, image processing, ML inference batches—to async serverless jobs and pipelines. Return fast, run slow, retry safely.
Last updated: 2026-04-20
- No HTTP timeout: pipelines run outside the request window
- Retries with step-level granularity—not all-or-nothing restarts
- Execution history for every job: duration, output, failures
- Same secrets and observability as your gateway API functions
Answer first
Direct answer
Long-running serverless jobs without HTTP timeouts. Inquir pipelines are not limited by HTTP request timeouts. Each pipeline step runs as a managed serverless function invocation with its own timeout budget—chain steps, fan out, wait for external callbacks, and retry failed steps independently.
When it fits
- Work genuinely takes longer than your HTTP runtime timeout—PDF generation, bulk API sync, ML batch inference, data archival.
- You want step-level retries instead of restarting from scratch on failure.
Tradeoffs
- Dropping a BullMQ consumer or SQS worker next to your API means a separate server, a separate deploy pipeline, a separate secret store, and a separate monitoring setup—before you write one line of job logic.
- Kubernetes Jobs help at scale but bring cluster overhead most products do not need for a handful of async workflows.
Workload and what breaks
Why HTTP timeouts kill slow work
Vercel Serverless Functions time out at 60s (Pro) or 300s (Enterprise). AWS Lambda caps at 900 seconds. Edge functions: 30 seconds. If your work takes longer—large file processing, bulk API calls, nightly data sync, multi-step ML pipelines—you are either racing the clock or using hacks like recursive invocations and background `fetch()`.
These hacks fail silently: a timeout mid-job leaves partial writes, no retries, and no trace in logs. You discover the problem when a customer files a ticket, not when the error happens.
Trade-offs
Why self-hosted worker queues add friction
Dropping a BullMQ consumer or SQS worker next to your API means a separate server, a separate deploy pipeline, a separate secret store, and a separate monitoring setup—before you write one line of job logic.
Kubernetes Jobs help at scale but bring cluster overhead most products do not need for a handful of async workflows.
How Inquir helps
Async pipelines that run outside the HTTP window
Inquir pipelines are not limited by HTTP request timeouts. Each pipeline step runs as a managed serverless function invocation with its own timeout budget—chain steps, fan out, wait for external callbacks, and retry failed steps independently.
The same workspace holds your HTTP routes, scheduled cron jobs, and long-running async pipelines. One secrets model, one execution history, one place to set alerts when a multi-hour job fails at step 4.
What you get
Long-running job patterns
Multi-step pipeline with per-step timeouts
Break a 3-hour ETL into 10-minute pipeline steps. If step 7 fails, retry step 7—not the whole job from scratch.
Fan-out and fan-in
Trigger N parallel pipeline branches, then aggregate results in a final step—parallel image resizing, bulk email, N-way API enrichment.
Human-in-the-loop pauses
Pause a pipeline step waiting for approval or external webhook before continuing—without holding a server process open.
Scheduled long jobs
Nightly ETL jobs that run for 2 hours, weekly report generation, monthly data archival—schedule via cron trigger, observe via execution history.
What to do next
How to move long-running work to Inquir pipelines
Acknowledge the HTTP request fast, hand off to a pipeline, run slow work in steps.
Return 202 from the HTTP handler
Accept the request, validate input, trigger background processing with global.durable.startNew(), and return a job reference immediately.
Implement each slow step as a function
Each pipeline step is a regular serverless function. Keep steps under 10 minutes; chain with dependsOn for sequencing.
Monitor via execution history
See every step run, retry, and output in the console. Set duration SLO alerts so long jobs surface before customers notice.
Code example
HTTP → pipeline handoff pattern
The HTTP handler validates input and returns 202 immediately. The pipeline step does the slow work. Use event.payload for pipeline context; event.previousOutput to read prior step results.
export async function handler(event) { const { reportId, format } = JSON.parse(event.body || '{}'); if (!reportId) return { statusCode: 400, body: JSON.stringify({ error: 'reportId required' }) }; // Return immediately — don't wait for the slow work const { instanceId: jobId } = await global.durable.startNew('generate-report', undefined, { reportId, format: format ?? 'pdf' }); return { statusCode: 202, body: JSON.stringify({ jobId, status: 'queued' }) }; }
export async function handler(event) { // Runs outside HTTP window — no timeout from the gateway const { reportId, format } = event.payload ?? {}; const data = await fetchAllReportData(reportId); // may take minutes const file = await renderReport(data, format); await storage.upload(file, `reports/${reportId}.${format}`); await notifyUser(reportId, file.url); return { reportId, size: file.byteLength, url: file.url }; }
When it fits
Use long-running serverless jobs when…
When this works
- Work genuinely takes longer than your HTTP runtime timeout—PDF generation, bulk API sync, ML batch inference, data archival.
- You want step-level retries instead of restarting from scratch on failure.
When to skip it
- Your work reliably finishes in under 10 seconds—keep it synchronous for simpler debugging.
FAQ
FAQ
How long can an Inquir pipeline step run?
Each step is a serverless function invocation with its own configurable timeout—significantly longer than HTTP request windows. Chain multiple steps for work that needs hours. See docs for current limits.
How do I track job status from the client?
Return a jobId from the 202 response. Poll a status endpoint that queries execution history, or have the final pipeline step post a webhook back to the client.
What if a step fails mid-job?
Configure retry count and delay per pipeline step. Failed steps retry independently without restarting earlier steps. Set alerts on step failure rates to catch recurring issues.