Skip to content
Docs

Tutorial: alerts for pipelines

A pipeline that fails at three in the morning should tell someone, and preferably tell a machine first. This tutorial builds an alert rule for a pipeline, sends it to Slack, checks that it fires, and then goes further: webhooks with your own payload, a function that reacts automatically, and a rule that pings a person when a run stops to wait for their approval.

What an alert rule is#

A rule watches runs of one kind, filters them by conditions, and when a run matches sends the alert to one or more channels. A cooldown stops the same rule from firing again for a while, so a step that fails in a loop produces one message rather than a hundred. Rules are managed on the Alerts page and come in three kinds:

  • Lambda function runs watch invocations of functions, whether they were called directly, through the gateway or from a pipeline.
  • Pipeline step runs watch each step of a pipeline as it executes, so you can target a pipeline by name and react to any of its steps failing or running too long.
  • Human gate fires when a run pauses at a human-in-the-loop step, which is how approvers learn that their decision is needed.

Where alerts go#

  • Slack: an incoming-webhook URL of a channel. The message shows the rule name, the pipeline or function, the status and a summary.
  • Webhook: a public HTTPS URL of your own with POST or PUT, optional headers, and a choice of payload: the plain alert object, a JSON with the trace attached, or a custom body with placeholders.
  • Function: invokes one of your functions with the alert as the event. Anything the function can do, restart, escalate, open a ticket, becomes the reaction.
  • Pipeline: starts a pipeline with the alert as its input, for reactions with several steps and their own approvals.

1. Create a rule for a pipeline#

Open Observability → Alerts and press New Alert Rule. Fill it in top to bottom:

  1. Rule name: something a colleague understands in Slack, for example Nightly ETL failed. Leave Enabled on.
  2. Rule kind: Pipeline step runs.
  3. Target pipelines: pick the pipeline. Leaving the list empty means every pipeline in the workspace, which is a fine default for a catch-all rule.
  4. Conditions: start with Status equals FAILED. Other useful ones: Duration greater than a number of milliseconds for steps that hang; Error message contains a phrase, to route one class of errors separately; Repeated failure with a count and a window, so the rule ignores a single flaky run and fires on the third failure in ten minutes; Log pattern matches a regular expression over the step logs.
  5. Channels: add Slack and paste the incoming-webhook URL of the channel where the alert should land. Add a second channel if you want a webhook or a function to react as well.
  6. Cooldown: how long, in milliseconds, the rule stays quiet after firing. Five minutes (300000) is a sensible start for a failure rule.

Save the rule. It starts watching the next run immediately; nothing needs to be redeployed.

2. Make it fire#

The cleanest test is a real failure. Open the pipeline, press Run (manual) and give it a payload that makes one step throw, for example a missing required field. The run goes red, and within seconds the Slack channel receives a message with the rule name, the pipeline, the step status and the error summary. Run it once more inside the cooldown: no second message, as intended.

Everything that fired is listed on the Alerts page as history, with the rule, the run it came from, the severity and the channels it was sent to. The run id links to the execution with its logs, so the path from a message in Slack to the failing line is two clicks.

3. Webhook payloads#

A webhook channel sends the alert record as JSON. With the default template the body is the alert itself; the json template adds a trace object with the run id, function name, status and error. Its shape:

webhook bodyjson
{
  "id": "…",
  "ruleId": "…",
  "ruleName": "Nightly ETL failed",
  "runId": "…",
  "functionId": "…",
  "functionName": "load-warehouse",
  "type": "status",
  "conditionTypes": ["status"],
  "severity": "error",
  "message": "…what matched, in words…",
  "details": { "…": "…" },
  "triggeredAt": "2026-09-05T02:00:03.000Z",
  "channels": ["webhook"]
}

The custom template lets you write the body yourself with placeholders such as {{alert.ruleName}}, {{alert.message}} and {{trace.status}}, which is how you match the format an incident tool or a chat bot already expects. Add headers for authentication. The URL must be public HTTPS; private addresses are refused.

4. Automate the reaction#

Notifying is the beginning. A Function channel calls one of your functions with the alert as the event, so a failed import can retry itself with a smaller batch or a stuck step can be cancelled and requeued. A Pipeline channel starts a whole pipeline from the alert, which is the right shape when the reaction needs several steps, its own retries, or a human decision before anything destructive happens.

5. Alerts when a run waits for a person#

A pipeline with a human-in-the-loop step pauses until someone approves, rejects or answers. Nobody watches an inbox all day, so create a second rule with the kind Human gate, optionally narrowed to the gate type, and send it to the approvers channel in Slack. The message carries the run and a link to the inbox, and the run continues the moment they act. How gates work, and how to build one, is the subject of the next tutorial.

Good habits#

  • Match the cooldown to the pipeline. A schedule that runs every minute needs a longer cooldown than a nightly job, otherwise one broken deploy becomes a wall of messages.
  • Route by severity. Alerts carry a severity from info to critical. Point warnings at a webhook that only logs, and criticals at the channel people actually read.
  • One catch-all, few specific rules. A workspace-wide failure rule with a generous cooldown catches everything; add narrow rules with error-message conditions for the cases that need a different reaction.
  • Pause, do not delete. The Enabled switch silences a rule during a migration or a known outage and keeps its history and settings for when it comes back.