> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flashcat.cloud/llms.txt
> Use this file to discover all available pages before exploring further.
# Alert Pipeline
> Cleanse, format, and preprocess data at the alert ingestion source
**Alert Processing Pipeline** allows you to **cleanse**, **transform**, and **filter** alert data at the moment it enters Flashduty On-call (integration layer). It acts like a data processing factory, ensuring alerts flowing into channels are standardized, clear, and valuable.
## Processing Order
After an alert event enters Flashduty, it passes through three stages in sequence:
| Stage | Description |
| :-------------------------------- | :------------------------------------------------------------------------------------------------------------ |
| **① Label Enhancement (Enrich)** | Dynamically generate or modify alert labels, such as querying CMDB via API to supplement business information |
| **② Alert Processing (Pipeline)** | Cleanse, transform, and filter alert data (the feature described in this document) |
| **③ Route Distribution (Route)** | Distribute events to corresponding channels based on alert attributes |
Since label enhancement runs before Pipeline, Pipeline match conditions **can reference labels added or modified during the enrichment stage**.
Recovery events (event\_status is Ok) bypass the alert processing Pipeline entirely and go directly into the alert merge flow. Therefore, discard, inhibition, rewrite, and other rules configured in Pipeline do not apply to recovery events.
After an event passes through the Pipeline, whether it merges into an existing alert is determined by the target channel's **Event Aggregation** setting (24-hour window by default, configurable from 1 to 1440 minutes, or can be turned off). See [Noise Reduction - Event Aggregation](/en/on-call/channel/noise-reduction#event-aggregation) for details.
## How It Works
Pipeline sits between **Label Enhancement** and **Route Distribution**. Its execution logic is as follows:
1. **Chain Processing**: You can configure multiple processing rules that execute sequentially **from top to bottom**
2. **Input/Output**: The result of a previous rule (e.g., modified title) can serve as input for the next rule
3. **Layer Positioning**: Pipeline operates at the **integration layer**. This means once rules take effect, all alerts ingested through this integration will be affected, regardless of which channel they're ultimately routed to
## Configuration Entry
Go to **Integration Center** => Select created integration => **Alert Processing** tab.
## Core Features and Scenarios
### Custom Severity
Flashduty On-call has built-in severity mapping for standard integrations (e.g., mapping Zabbix `High` to `Critical`). But if default rules don't meet your needs, you can override them via Pipeline.
**Scenario**: `Warning` level alerts in the monitoring system are actually critical for core payment business and need to be upgraded to `Critical` to trigger phone notifications.
| Config Item | Value |
| :---------- | :---------------------------------------------------------------- |
| Condition | `labels.service` equals `payment` AND `severity` equals `Warning` |
| Action | Update severity to `Critical` |
### Information Rewrite
Convert cryptic machine language to human-readable business language using template syntax.
**Scenario**: Original title contains lots of IDs without business meaning.
| Item | Content |
| :----------------- | :------------------------------------------------------------------------------ |
| Original Title | `[Problem] CPU Load High on i-12345678` |
| New Title Template | `[TPL]{{.Labels.env}} Environment - {{.Labels.service}} Service CPU Load Alert` |
| Result | `Production Environment - Order Service CPU Load Alert` |
**Scenario**: Automatically append Runbook links or dashboard URLs in description to assist quick troubleshooting.
| Config Item | Value |
| :------------- | :---------------------------------------------------------------------------- |
| Action | Update alert description |
| Append Content | `Grafana Dashboard: https://grafana.corp.com/d/cpu?var-host={{.Labels.host}}` |
### Alert Discard
Discard directly before data storage, leaving no records. This is similar to "Exclusion Rules" in channels but takes effect earlier.
| Scenario | Advantage |
| :------------------------------------------- | :------------------------------------------------------ |
| Frequent restart alerts from dev environment | Cleanse at source |
| Known harmless errors (like "NTP offset") | Reduce load on subsequent routing and storage resources |
### Alert Inhibition
Pipeline's inhibition feature is similar to channel inhibit rules, both supporting dependency-based inhibition based on source alert, target alert, and correlation conditions. Note that Pipeline inhibition only matches source alerts **within the same integration**, while channel inhibition matches across all alerts in the channel.
| Comparison | Pipeline Inhibition | Channel Inhibition |
| :--------------- | :--------------------------------------------------------------------------------------------- | :------------------------------------------- |
| Effective Layer | Integration layer | Channel layer |
| Match Scope | Only matches active alerts within the same integration | Matches all active alerts within the channel |
| Use Case | **Global** inhibition logic, like inhibiting all alerts from a datacenter after network outage | Inhibit rules for specific channels |
| Plan Requirement | Requires Standard plan or above | Requires Standard plan or above |
When an entire datacenter loses network, all alerts from that datacenter (regardless of business line) should be inhibited. Configuring one rule at the integration layer is much more efficient than configuring separately in dozens of channels.
Inhibition only matches source alerts that have been active within the last 10 minutes (600 seconds). If a source alert was triggered more than 10 minutes ago and has not been updated since, it will not be used for inhibition matching. Ensure your source alerts remain active within this time window.
## Reference Historical Data
When editing alert processing rules, the right side of the page provides a **Related Alerts** panel showing historical alert data ingested through the current integration. You can write processing rules while referencing actual alert content, ensuring that condition expressions and action configurations meet expectations.
Click the **Reference Historical Data** button in the upper right of the editing page to open or close the related alerts panel. The panel supports drag-to-resize, allowing you to flexibly switch between the editing area and data preview.
When configuring Pipeline for the first time, we recommend opening the related alerts panel and referencing real alert labels and field values to write matching conditions, avoiding rule misses due to inconsistent field names.
## Reference Syntax
When rewriting titles or descriptions, you can use Go Template syntax to reference internal alert variables. Proper use of variable references makes your alert notifications more dynamic and informative.
| Variable | Description | Example |
| :------------------------ | :---------------------------- | :--------------------------------------- |
| `[TPL]{{.Labels.xxx}}` | Reference specific label | `[TPL]{{.Labels.host}}` |
| `[TPL]{{.Title}}` | Reference current title | `[TPL][Forwarded] {{.Title}}` |
| `[TPL]{{.Description}}` | Reference current description | `[TPL]Details: {{.Description}}` |
| `[TPL]{{.EventSeverity}}` | Reference current severity | `[TPL]Current Level: {{.EventSeverity}}` |
Before configuring Pipeline, we recommend observing raw alert data for a period to identify which fields (Labels) are stable and which need cleansing.
## Further Reading
Dynamically generate or modify alert labels
Learn condition matching syntax
Distribute alerts to different channels
Inhibit alerts at the channel layer