Skip to content
Docs

Tutorial: alerts for pipelines

A pipeline that fails at three in the morning should tell someone, and preferably tell a machine first. This tutorial builds an alert rule for a pipeline, sends it to Slack, checks that it fires, and then goes further: webhooks with your own payload, a function that reacts automatically, and a rule that pings a person when a run stops to wait for their approval.

What an alert rule is#

A rule watches runs of one kind, filters them by conditions, and when a run matches sends the alert to one or more channels. A cooldown stops the same rule from firing again for a while, so a step that fails in a loop produces one message rather than a hundred. Rules are managed under Alerts → Rules in the sidebar and come in three kinds:

  • Function runs watch invocations of functions, whether they were called directly, through the gateway or from a pipeline.
  • Pipeline step runs watch each step of a pipeline as it executes, so you can target a pipeline by name and react to any of its steps failing or running too long.
  • Human gate (inbox pause) fires when a run pauses at a human-in-the-loop step, which is how approvers learn that their decision is needed.

Where alerts go#

  • Slack: an incoming-webhook URL of a channel. The message shows the rule name, the pipeline or function, the status and a summary.
  • Webhook: a public HTTPS URL of your own with POST or PUT, optional headers, and a choice of payload: the plain alert object, a JSON with the trace attached, or a custom body with placeholders. The console form does not offer this channel yet; add it to a rule through the API (/api/observability/alerts/rules).
  • Function: invokes one of your functions with the alert as the event. Anything the function can do, restart, escalate, open a ticket, becomes the reaction.
  • Pipeline: starts a pipeline with the alert as its input, for reactions with several steps and their own approvals.

1. Create a rule for a pipeline#

Open Alerts in the sidebar, press Create rule, choose Functions and pipelines and press Continue. Fill in the form top to bottom:

  1. Rule name: something a colleague understands in Slack, for example Nightly ETL failed. Leave Enabled on.
  2. What should this rule listen for?: Pipeline step runs.
  3. Target pipelines: pick the pipeline. Leaving the list empty means every pipeline in the workspace, which is a fine default for a catch-all rule.
  4. Conditions: press Add and start with Status · Equals · FAILED. Other useful ones: Duration (ms) with Greater than and a number of milliseconds for steps that hang; Error message with Contains and a phrase, to route one class of errors separately; Repeated failure with a count and a window, so the rule ignores a single flaky run and fires on the third failure in ten minutes. A log-pattern condition, a regular expression over the step logs, is available through the API.
  5. Channels: press Add, choose Slack and paste the incoming-webhook URL of the channel where the alert should land. Add a second channel if you want a function or a pipeline to react as well.
  6. Cooldown (ms): how long, in milliseconds, the rule stays quiet after firing. Five minutes (300000) is a sensible start for a failure rule.

Press Create rule. It starts watching the next run immediately; nothing needs to be redeployed.

2. Make it fire#

The cleanest test is a real failure. Open the pipeline, press Run (manual) and give it a payload that makes one step throw, for example a missing required field. The run goes red, and within seconds the Slack channel receives a message with the rule name, the pipeline, the step status and the error summary. Run it once more inside the cooldown: no second message, as intended.

Everything that fired is listed under Alerts → History, with the rule, the run it came from, the severity and the channels it was sent to. The run id links to the execution with its logs, so the path from a message in Slack to the failing line is two clicks.

3. Webhook payloads#

A webhook channel sends the alert record as JSON. With the default template the body is the alert itself; the json template adds a trace object with the run id, function name, status and error. Its shape:

webhook bodyjson
{
  "id": "…",
  "ruleId": "…",
  "ruleName": "Nightly ETL failed",
  "runId": "…",
  "functionId": "…",
  "functionName": "load-warehouse",
  "type": "status",
  "conditionTypes": ["status"],
  "severity": "error",
  "message": "…what matched, in words…",
  "details": { "…": "…" },
  "triggeredAt": "2026-09-05T02:00:03.000Z",
  "channels": ["webhook"]
}

The custom template lets you write the body yourself with placeholders such as {{alert.ruleName}}, {{alert.message}} and {{trace.status}}, which is how you match the format an incident tool or a chat bot already expects. Add headers for authentication. The URL must be public HTTPS; private addresses are refused.

4. Automate the reaction#

Notifying is the beginning. A Function channel calls one of your functions with the alert as the event, so a failed import can retry itself with a smaller batch or a stuck step can be cancelled and requeued. A Pipeline channel starts a whole pipeline from the alert, which is the right shape when the reaction needs several steps, its own retries, or a human decision before anything destructive happens.

5. Alerts when a run waits for a person#

A pipeline with a human-in-the-loop step pauses until someone approves, rejects or answers. Nobody watches an inbox all day, so create a second rule with the kind Human gate (inbox pause), optionally narrowed with Human gate type, and send it to the approvers channel in Slack. The message carries the run and a link to the inbox, and the run continues the moment they act. How gates work, and how to build one, is the subject of the next tutorial.

Good habits#

  • Match the cooldown to the pipeline. A schedule that runs every minute needs a longer cooldown than a nightly job, otherwise one broken deploy becomes a wall of messages.
  • Route by severity. Alerts carry a severity from info to critical. Point warnings at a webhook that only logs, and criticals at the channel people actually read.
  • One catch-all, few specific rules. A workspace-wide failure rule with a generous cooldown catches everything; add narrow rules with error-message conditions for the cases that need a different reaction.
  • Pause, do not delete. Disable in the rules list silences a rule during a migration or a known outage and keeps its history and settings for when it comes back.