Scheduled pipelines with run history, retries, and logs
Add a cron trigger to a pipeline for recurring work: expressions validated at save time, execution history beside HTTP invocations, configurable retries, and the same secrets and logging model as your API functions—no separate scheduling service or VPS crontab.
Last updated: 2026-10-11
Answer first
Direct answer
Scheduled pipelines with run history, retries, and logs. Each scheduled job is a pipeline with a cron trigger. The platform validates the cron expression at save time, tracks next-run-at per pipeline, and creates an invocation record for every execution—queryable without SSH. A Postgres advisory-lock leader election means only one node fires each schedule, even when you run multiple workers.
When it fits
- Nightly ETL, hourly sync, weekly reports, certificate rotation
- Any recurring task that needs visible run history and retries
Tradeoffs
- When job state lives on a single host, every deploy might reset the schedule, every SSH session is a potential foot-gun, and every rotation of the VPS means re-reading a 2-year-old runbook.
- Secrets stored in
.envfiles on the server are audit nightmares; crontab entries are not version-controlled with your application code.
Workload and what breaks
Silent failures in traditional scheduled jobs
- VPS crontab: output routed to root mail nobody reads; no retry on failure
- systemd timers: better reliability, but logs scattered, secrets manual, no history
- Kubernetes CronJob: correct primitives but cluster overhead for small teams
Scheduled jobs fail silently more often than any other backend primitive. The job runs on a VPS, the script exits non-zero, the error goes to a mail spool, and nobody knows until data is 3 days stale.
Trade-offs
Why crontab-on-a-server doesn't scale past one engineer
When job state lives on a single host, every deploy might reset the schedule, every SSH session is a potential foot-gun, and every rotation of the VPS means re-reading a 2-year-old runbook.
Secrets stored in .env files on the server are audit nightmares; crontab entries are not version-controlled with your application code.
How Inquir helps
Scheduled pipelines as first-class serverless
Each scheduled job is a pipeline with a cron trigger. The platform validates the cron expression at save time, tracks next-run-at per pipeline, and creates an invocation record for every execution—queryable without SSH. A Postgres advisory-lock leader election means only one node fires each schedule, even when you run multiple workers.
The scheduler polls every 30 seconds (minimum 1-minute interval), so fires land within ~30s of the scheduled time. Scheduled jobs share workspace secrets and the same observability stack as HTTP routes—nightly ETL, weekly reports, hourly sync all show up in execution history next to webhook and API invocations.
Compare
Crontab vs systemd timers vs Kubernetes CronJob vs Inquir
Teams pick a scheduler based on ops maturity and where the rest of the stack lives. This table contrasts four common approaches for recurring backend work—not edge cron or sub-second tickers.
| Dimension | crontab | systemd timers | Kubernetes CronJob | Inquir scheduled pipelines |
|---|---|---|---|---|
| Setup & ops burden | Edit user crontab on a host; no expression validation at save time | Unit + timer files; more reliable firing, still host-level packaging | YAML manifests and a running cluster—even for one nightly job | Cron on a pipeline trigger; expression validated when you save |
| Run history | Redirect stdout to a log file or root mail—no structured audit trail | journalctl per unit; grep across services when jobs multiply | Pod logs plus Job status; retention depends on cluster logging stack | Invocation records beside HTTP executions—queryable without SSH |
| Retries on failure | None—next fire is the only implicit retry | OnFailure= can restart the unit; no per-run retry policy in product UI | Job backoffLimit; tuned in manifest, not beside your API history | Per-step retry count and delay on the pipeline |
| Secrets | .env on the server or inline in shell—shared across jobs on the host | EnvironmentFile units; rotation is still manual per host | Kubernetes Secrets—cluster-native but separate from app gateway auth | Workspace secrets shared with HTTP routes—no parallel server .env |
| Isolation between jobs | One UNIX user and interpreter env—dependency conflicts are common | Can run as different users; still one machine, shared kernel | Pod per run—strong isolation when cluster is already justified | Container per invoke—same isolation model as synchronous HTTP handlers |
| Version control with app code | Crontab lines often live outside deploy pipelines | Unit files can live in git; drift between hosts still happens | Manifests in git/GitOps—excellent when K8s is already the control plane | Function bundles and pipeline config versioned with platform deploy |
| Multi-step workflows | Shell scripts chaining commands—opaque failure points | Separate units or bash in ExecStart—manual orchestration | Single container per CronJob; DAGs need Argo/Workflows or similar | Pipeline graphs with edges—extract, transform, load as function nodes |
| Overlap / long runs | Manual flock or hope—overlapping runs corrupt shared state | Concurrency settings per unit—still DIY in handler logic | concurrencyPolicy Forbid/Replace on the CronJob spec | Skip-if-running guards and idempotency keys in handler or pipeline design |
| Best fit | One VPS, one engineer, scripts that already work and rarely fail | Linux ops teams standardized on systemd who accept journal diving | Products already on Kubernetes with GitOps and cluster logging | Scheduled pipelines plus REST on one platform—history and retries without cluster YAML |
What you get
Pipeline scheduling features
Cron expression validation
Expressions are validated when you save the pipeline—malformed entries fail immediately, not silently on the next tick.
Run history and logs
Every scheduled execution creates an invocation record: start time, duration, exit status, step outputs. Queryable without SSH.
Retries and overlap guards
Configure retry count and delay per step. Add idempotency keys for overlap protection when long jobs run past their interval.
Shared secrets
Scheduled jobs use the same workspace secrets as HTTP routes—no parallel .env files on a server.
What to do next
How to schedule a pipeline on Inquir
Write the job handler
Standard serverless function. Use environment variables for secrets; return structured output per step.
Add a pipeline schedule
In the visual pipeline editor, connect a cronTrigger node to the function step and enter the cron expression. Save the pipeline to validate its schedule.
Monitor run history
Open execution history to see every run, duration, and output without SSH access.
Code example
Nightly sync job with watermark
Scheduled handler reads cursor from environment, fetches incremental updates, upserts idempotently, and returns new cursor for the next run.
export async function handler(event) { // event.trigger?.type === 'schedule' when fired by a cronTrigger node const since = process.env.SYNC_CURSOR ?? new Date(Date.now() - 86_400_000).toISOString(); const records = await source.fetchUpdatedSince(since); if (records.length === 0) return { synced: 0, cursor: since }; await destination.upsertBatch(records); // idempotent by record ID const newCursor = records.at(-1)?.updatedAt ?? since; // Store cursor for next run (update env var or external store) return { synced: records.length, cursor: newCursor }; }
When it fits
Use scheduled pipelines for…
When this works
- Nightly ETL, hourly sync, weekly reports, certificate rotation
- Any recurring task that needs visible run history and retries
When to skip it
- Sub-minute scheduling—cron minimum interval is 1 minute; the scheduler fires within ~30s of the scheduled time but cannot guarantee second-level precision
FAQ
FAQ
Can one function serve HTTP and a scheduled pipeline?
Yes. Use the same function ID in a pipeline function step and a gateway HTTP route. The cron trigger starts the pipeline, which calls that step.
What timezone do cron expressions use?
Use UTC for production schedules unless your requirements are strictly wall-clock business hours. Document the timezone assumption in the pipeline name.
What is the minimum cron interval?
One minute. The scheduler polls every 30 seconds, so actual fire time can be up to 30 seconds after the scheduled minute. Never rely on intervals shorter than 1 minute or on second-level precision.
What if a run takes longer than the interval?
Use idempotency keys, distributed locks, or skip-if-running guards inside the handler so overlapping firings do not corrupt shared state. For idempotent jobs, a watermark cursor makes overlap safe by design.
Can I test the job before the next scheduled run?
Yes—invoke the function directly from the gateway or the console. That tests the handler, not the pipeline schedule or the whole workflow.