Serverless job queue: drop the Redis worker fleet
BullMQ needs Redis, a worker process, and a retry strategy. SQS needs IAM, a consumer Lambda, and a dead-letter queue config. Inquir pipelines give you producer/consumer semantics—enqueue a job, run it in an isolated container, retry failed steps—without provisioning or operating any queue infrastructure.
Last updated: 2026-06-28
Answer first
Direct answer
Serverless job queue: drop the Redis worker fleet. Inquir is not a high-throughput message broker with strict ordering, but you do not need to operate one. Pipelines plus a durable background-jobs queue (Postgres-backed retries, exponential backoff, and a dead-letter queue) remove the operational burden of running a queue—retry policy per step, per-step run history, and isolated container execution—so you ship job logic instead of queue infrastructure.
When it fits
- You want enqueue + retry + run history without operating Redis, SQS, or a worker fleet
- Background jobs should share secrets and observability with your HTTP API surface
Tradeoffs
- In-memory queues inside your API process lose jobs on restart and cannot scale horizontally without duplicate delivery. The worker process is the most fragile component: it needs to stay running, drain on deploy, and handle crashes without losing in-flight work.
- Calling SQS from every handler spreads IAM credentials, retry logic, and idempotency key management across the codebase. Each new job type recreates the same boilerplate.
Workload and what breaks
What a production job queue actually costs to operate
- Queue backend: Redis, SQS, or RabbitMQ—provisioned, monitored, and on the critical path for every job
- Worker fleet: persistent processes that consume the queue, scale with load, and drain safely on deploy
- Retry and backoff policy: exponential backoff, max attempts, and manual inspection when jobs exhaust retries
- Observability: a separate dashboard to see queue depth, stalled jobs, and failure reasons
Job queues solve real problems—decoupling producers from consumers, absorbing traffic spikes, retrying flaky work. But the queue itself becomes infrastructure you operate before the first business job runs.
Trade-offs
Why DIY queues add operational debt without solving the core problem
In-memory queues inside your API process lose jobs on restart and cannot scale horizontally without duplicate delivery. The worker process is the most fragile component: it needs to stay running, drain on deploy, and handle crashes without losing in-flight work.
Calling SQS from every handler spreads IAM credentials, retry logic, and idempotency key management across the codebase. Each new job type recreates the same boilerplate.
How Inquir helps
Pipelines as producer/consumer infrastructure without the queue operations
Inquir is not a high-throughput message broker with strict ordering, but you do not need to operate one. Pipelines plus a durable background-jobs queue (Postgres-backed retries, exponential backoff, and a dead-letter queue) remove the operational burden of running a queue—retry policy per step, per-step run history, and isolated container execution—so you ship job logic instead of queue infrastructure.
The producer calls global.durable.startNew(jobName, undefined, payload) from any function. The consumer is the pipeline step handler, run by a managed worker pool (10 concurrent per function, 50 global, internal queue depth 100). No Redis connection string, no worker systemd unit, no queue drain script on deploy.
Compare
Queue operational ownership comparison
What you own and operate for each approach—not a feature comparison, an operational one.
| Ownership area | BullMQ + Redis | AWS SQS + Lambda | Inquir Pipelines |
|---|---|---|---|
| Queue infrastructure | Redis cluster (you provision) | SQS queue (AWS managed) | None—built into pipeline |
| Worker process | BullMQ workers (you run) | Lambda consumer (you configure) | Pipeline steps (platform runs) |
| Retry config | Per-job in worker code | Redrive policy + DLQ | Per-step in pipeline config |
| Run history / observability | Redis keys (short TTL) | CloudWatch + DLQ inspection | Execution records (30-day retention) |
| Secrets model | Separate from HTTP routes | IAM + Secrets Manager | Shared workspace secrets |
| Delivery guarantee | At-least-once with Redis | At-least-once with SQS | At-least-once (no guaranteed ordering) |
What you get
Job queue patterns on Inquir
Enqueue from any producer
HTTP 202, verified webhook, cron schedule, or another pipeline step—one enqueue call (global.durable.startNew) regardless of entry point.
Per-step retry with backoff
Configure retry count and delay per pipeline step. Failed steps retry without re-running earlier completed steps—equivalent to job retry options without a separate retry queue.
Run history as execution record
Every enqueue creates a run record: input payload, step outputs, duration, retry count, and failure reason. Inspectable from the console without querying Redis or checking DLQ depth.
Fan-out without a second queue
One job step calls global.durable.startNew() N times to spawn parallel child pipelines. No separate queue product or topic fanout configuration.
What to do next
How to replace a BullMQ or SQS queue with Inquir pipelines
Write the consumer handler
Export a function that reads event.payload and returns structured output—the consumer side. Make it idempotent: check a dedupe key before writing side effects.
Enqueue from the producer
Replace your queue.add() or sqs.sendMessage() call with global.durable.startNew(jobName, undefined, payload) and return immediately.
Configure retry policy and monitor
Set retry count and backoff in pipeline step config. Watch execution history for stalled runs; alert on failure rate thresholds.
Code example
Producer → consumer without Redis or SQS
The producer validates input and enqueues in one call. The consumer handler runs in an isolated container with platform-managed retry.
export async function handler(event) { const { exportId, format } = JSON.parse(event.body || '{}'); if (!exportId) return { statusCode: 400, body: JSON.stringify({ error: 'exportId required' }) }; // Replace: queue.add('run-export', { exportId, format }) const { instanceId: jobId } = await global.durable.startNew('run-export', undefined, { exportId, format: format ?? 'csv' }); return { statusCode: 202, body: JSON.stringify({ jobId, status: 'queued' }) }; }
export async function handler(event) { const { exportId, format } = event.payload ?? {}; // Idempotency: skip if already exported const existing = await db.findExport(exportId); if (existing) return { exportId, fileUrl: existing.url, skipped: true }; const rows = await fetchExportRows(exportId); const fileUrl = await writeExport(rows, format); await db.saveExport(exportId, fileUrl); await notifyExportReady(exportId, fileUrl); return { exportId, fileUrl, rowCount: rows.length }; }
When it fits
When Inquir pipelines replace a job queue
When this works
- You want enqueue + retry + run history without operating Redis, SQS, or a worker fleet
- Background jobs should share secrets and observability with your HTTP API surface
When to skip it
- Sub-millisecond queue latency with strict FIFO ordering across millions of jobs per second—use a dedicated message broker for that throughput and ordering guarantee
FAQ
FAQ
Does Inquir guarantee exactly-once delivery?
No. Pipeline invocations are at-least-once. Make consumer handlers idempotent—check a dedupe key (e.g. exportId, jobId) before writing side effects so retries are safe.
Is there a dead-letter queue?
Yes. The durable background-jobs queue dead-letters a job after it exhausts its retry attempts, recording the last error so you can inspect and replay poison messages instead of losing them. Pipeline step failures are additionally retained in execution history for manual replay from the stored payload.
Can I enqueue from Python or Go handlers?
global.durable.startNew() is the Node.js SDK method. From Python or Go, POST to the pipeline trigger URL with the same payload shape—the trigger API is HTTP-accessible from any runtime.
How does this compare to BullMQ specifically?
BullMQ runs jobs in worker processes consuming a Redis queue—you operate Redis, scale workers, and manage the DLQ. Inquir pipeline steps run in isolated containers managed by the platform. You write the consumer handler; the platform handles scheduling, retries, and run records.