> ## Documentation Index
> Fetch the complete documentation index at: https://docs.flashcat.cloud/llms.txt
> Use this file to discover all available pages before exploring further.

# Prometheus Integration Guide

> Push Prometheus alert events to Flashduty through AlertManager using webhooks. When an alert is triggered, it sends a trigger event to Flashduty, and when the alert recovers, it sends a recovery event.

Push Prometheus alert events to Flashduty through AlertManager using webhooks. When an alert is triggered, it sends a trigger event to Flashduty, and when the alert recovers, it sends a recovery event.

<div class="hide">
  ## Requirements

  ***

  ### In AlertManager

  * You must have permission to modify the AlertManager configuration file.
  * Your AlertManager server must be able to access api.flashcat.cloud domain to push alerts to the internet.

  ## Supported Versions

  ***

  This guide is compatible with **Alertmanager 0.16.0 and above**.

  ## Setup Steps

  ***

  ### In Flashduty

  You can obtain an integration push URL through either of these two methods:

  #### Using Private Integration

  Choose this method when you don't need to route alert events to different channels. It's simpler.

  <details>
    <summary>Expand</summary>

    1. Go to the Flashduty console, select **channel**, and enter a channel's details page
    2. Select the **Integrations** tab, click **Add Integration**, and enter the integration page
    3. Select **Prometheus** integration, click **Save** to generate a card
    4. Click the generated card to view the **push URL**, copy it for later use, and you're done
  </details>

  #### Using Shared Integration

  Choose this method when you need to route alert events to different channels based on the alert payload.

  <details>
    <summary>Expand</summary>

    1. Go to the Flashduty console, select **Integration Center=>Alert Events** to enter the integration selection page
    2. Select **Prometheus** integration:
       * **Integration Name**: Define a name for this integration
    3. Configure the default route and select the corresponding channel (after the integration is created, you can go to `Route` to configure more routing rules)
    4. Click **Save** and copy the newly generated **push URL** for later use
    5. Done
  </details>
</div>

### In AlertManager

#### Step 1: Configure Alertmanager

<div class="md-block">
  1. Log in to your Alertmanager instance

  2. Locate and open the configuration file, typically alertmanager.yml in the Alertmanager deployment root directory

  3. In the receivers configuration section, add a Flashduty webhook type receiver as follows:

     ```
     receivers:
     - name: 'flashcat'
       webhook_configs:
       - url: '<Your Integration Push URL>'
         send_resolved: true

     ```

     Replace the url parameter value with your integration push URL, ensuring the integration\_key is included in the query string parameters.

     If you need to request Flashduty through a proxy, you can additionally set the http\_config proxy\_url parameter:

     ```
     receivers:
     - name: 'flashcat'
       webhook_configs:
       - url: '<Your Integration Push URL>'
         send_resolved: true
         http_config:
         proxy_url: 'http://proxyserver:port'

     ```

  4. In the route configuration section, change the default route to specify the newly configured webhook receiver:

     ```route config theme={null}
     route:
       receiver: 'flashcat'
     ```

     If you don't want to affect existing push channels, you can add the receiver to a route's subroute:

     ```route config theme={null}
     route:
      receiver: 'feishu'
      - routes:
        receiver: 'flashcat'
     ```

  5. Save the configuration file

  6. Reload the configuration (by sending a SIGHUP signal to the process or POST request to /-/reload api)

  7. Done
</div>

#### Step 2: Configure Timestamp

By default, the system uses the current time as the event trigger time. If you want to customize the time, you can set an additional timestamp field to identify the exact time of each alert.

<div class="md-block">
  1. Log in to your Prometheus Server instance

  2. Open the alert rules configuration file

  3. For each alert rule, modify the annotations section by adding a timestamp field:

     ```
     annotations:
       timestamp: '{{ with query "time()" }}{{ . | first | value }}{{ end }}'
       ...
     ```

  4. Save the configuration file

  5. Reload the configuration (by sending a SIGHUP signal to the process or POST request to /-/reload api)

  6. Done
</div>

## Severity Mapping

***

The system extracts `severity`, `priority`, and `level` from alert event labels in that order. The corresponding value will be used as Prometheus's own alert severity. If none are found, the system automatically sets the Prometheus alert severity to `Warning`.

Prometheus to Flashduty severity mapping:

| Prometheus   | Flashduty | Status   |
| ------------ | --------- | -------- |
| critical     | Critical  | Critical |
| warning      | Warning   | Warning  |
| warn         | Warning   | Warning  |
| info         | Info      | Info     |
| acknowledged | Info      | Info     |
| unknown      | Info      | Info     |
| unk          | Info      | Info     |
| ok           | Ok        | Ok       |

## FAQ

***

<details>
  <summary>Why am I not receiving alerts in Flashduty?</summary>

  #### In Flashduty

  1. Check if the integration shows **Latest Event Time**. If not, Flashduty hasn't received any pushes - prioritize checking the AlertManager section.
  2. If you're using a **Shared Integration**, first confirm if you've configured **Routing Rules**. Without routing rules, the system will reject new pushes as there's no channel to receive your alerts. In this case, simply configure routing rules to your desired channel.

  #### In AlertManager

  1. First, confirm if AlertManager has generated new alerts. If not, wait for new alerts to trigger and verify again.
  2. Open the AlertManager configuration file. If you've set up subroutes, ensure your routing is correct (e.g., if previous routes set continue, AlertManager will skip matching subsequent subroutes. We recommend setting only one default route to Flashduty). Also verify that the callback URL matches the integration push URL exactly. If they don't match, modify the **Alert Rules** and verify again.
  3. If they match, confirm that your AlertManager instance can access api.flashcat.cloud domain. If not, you need to enable internet access or specifically allow access to Flashduty's domain.
  4. If network access is fine, continue troubleshooting AlertManager by checking for relevant error logs.

  If you still can't identify the root cause after following these steps, please contact us directly.
</details>