ElasticSearch

​

Core Concepts

​

1. Threshold Evaluation Mode

​

Configuration

1. Query Statement: Write SQL aggregate query, returning value columns and (optional) grouping columns.

• Example: Count error log quantity by service in the last 5 minutes.
  Copy

  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  

2. Field Mapping:

• Label Fields: Fields used to distinguish different alert objects. In the above example, it’s service_name. Can be left empty; Monitors will automatically treat all fields except value fields as label fields.
• Value Fields: Numeric fields used for threshold evaluation. In the above example, it’s error_cnt.

3. Threshold Conditions:

• Use $A.field_name to reference values.
• Example: Critical: $A.error_cnt > 50, Warning: $A.error_cnt > 10.
• Shorthand: If only one value field is configured, you can directly use $A, like $A > 50.

​

How It Works

​

Recovery Logic

SELECT network_host, interface, status FROM "network-status-*" 
WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
  AND network_host = '${network_host}' 
  AND interface = '${interface}' 
  AND status = 'UP'

​

2. Data Exists Mode

​

Configuration

1. Query Statement: Use HAVING clause in SQL to directly filter out anomalous data.

• Example: Directly query services with error count exceeding 50.
  Copy

  SELECT service_name, count(*) AS error_cnt 
  FROM "app-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES AND log_level = 'ERROR'
  GROUP BY service_name
  HAVING count(*) > 50
  

2. Field Mapping:

• In this mode, label fields and value fields are optional. If both are left empty, the engine will treat all fields in query results as label fields, which can be referenced in rule notes.

​

Recovery Logic

• Recovery When Data Disappears: When SQL query result is empty (i.e., no longer satisfies HAVING condition), the engine determines incident recovery. This is the most commonly used recovery method.
• Recovery Query:
  • Scenario: Sometimes “no data found” doesn’t mean recovery (might be log collection down), or need stricter recovery conditions (like no errors for N consecutive minutes).
  • Configuration: Write an independent SQL statement for recovery evaluation. As long as that query can find data, the incident is considered recovered.
  • Variable Support: Supports using ${label_name} in recovery SQL to reference alert event label values for precise recovery detection.

​

Pros and Cons Analysis

​

3. No Data Mode

​

Configuration

1. Query Statement: Write a SQL query that is expected to continuously return data.

• Example: Query log reporting heartbeat from all hosts.
  Copy

  SELECT host_name 
  FROM "heartbeat-logs-*" 
  WHERE "@timestamp" > now() - INTERVAL 5 MINUTES 
  GROUP BY host_name
  

2. Evaluation Rules:

• The engine periodically executes this SQL.
• If a host_name appeared in previous cycles but no longer appears in current cycle (and N consecutive cycles) query results, triggers “No Data” alert.
• Note: This is opposite to Data Exists mode. Data Exists means “alert when data found”, No Data means “alert when data not found”.

​

Recovery Logic

​

4. Use Case

• Main Alert Condition: Use Threshold mode, SQL statement counts ERROR logs in the last 5 minutes, configure threshold conditions.
• Related Query: Configure a related query, SQL statement queries the most recent ERROR log, using ${service_name} and other variables to limit to specific service.
• Rule Notes Description: Reference related query results in alert rule’s notes description, using $relates variable to render the original log.