Loki Alert Rule Configuration

This document details how to configure Loki data source alert rules in the Monitors alert engine. Monitors supports Loki's LogQL query syntax, enabling aggregation analysis of log data and alert triggering.

Core concepts

Loki's query language LogQL is divided into two categories:

Log queries: Return log line content (Stream).

Metric queries: Count or aggregate logs, such as using count_over_time function to return values (Vector).

1. Threshold mode

This mode is suitable for scenarios that require multi-level threshold evaluation (such as Info/Warning/Critical) on log aggregation values.

Configuration

Query statement (LogQL): Write a LogQL that returns a numeric vector (select "Statistics" query mode).

Example: Count the number of logs containing the error keyword in the mysql job over the last 5 minutes.

count_over_time({job="mysql"} |= "error" [5m])

Threshold conditions:

Critical: $A > 50 (more than 50 error logs in 5 minutes)

Warning: $A > 10 (more than 10 error logs in 5 minutes)

How it works

The engine executes the LogQL query and retrieves time series data (Vector) with labels. The engine iterates through each series, extracts the value, and compares it against the configured threshold expressions.

Recovery logic

Automatic recovery: When the query result value falls below the threshold, it automatically recovers.

Specific recovery condition: Can be configured as $A < 5 to avoid oscillation near the threshold.

Recovery query:

Supports configuring a separate LogQL for recovery evaluation; recovery is triggered as long as data is found.

Supports ${label_name} variable substitution.

Example: Alert on error logs, recover on specific recovery logs count_over_time({job="mysql"} |= "recovered" [5m]).

2. Data exists mode

This mode is suitable for scenarios where you prefer to write filter conditions directly in LogQL, or only care about "whether abnormal data exists". This mode is recommended for log anomaly detection alerts.

Configuration

Query statement (LogQL): Write a LogQL containing comparison operators that returns only data meeting the conditions.

Example: Directly filter out services with error rates exceeding 5%.

count_over_time({job="ingress"} |= "error-code-500" [5m]) / count_over_time({job="ingress"} [5m]) * 100 > 5

Evaluation rule: An alert is triggered as long as the LogQL query returns data.

Pros and cons

Pros: Computation logic is pushed down to the Loki server, reducing data transfer.

Cons: Cannot distinguish alert levels; can only trigger a single level of alert.

Recovery logic

Recover when data disappears: When the LogQL query result is empty (i.e., no longer meets the > 5 condition), recovery is determined.

Recovery query: Supports configuring additional query statements to assist in determining recovery status.

3. No data mode

This mode is used to monitor whether the log reporting pipeline is interrupted, or whether logs that should be continuously generated have stopped.

Configuration

Query statement (LogQL): Write a query that should always have data.

Example: Count the log reporting rate for all hosts.

rate({job="node-logs"} [1m])

Evaluation rule: If a Series (uniquely identified by labels, such as instance="host-1") existed in previous cycles but cannot be found in the current and consecutive N cycles, a "no data" alert is triggered.

Typical applications

Monitor whether collection agents like Promtail/Fluentd have stopped working.

Monitor whether critical business logs (such as order creation logs) have been abnormally interrupted.

4. Get original logs when alerting

You can get original logs through related queries when alerting. However, it is usually not recommended to get too many; just get 1 as a log sample to include in the alert message.

The results of related queries can be rendered in the "Note description", example:

{{- if eq $status "firing" }}
error log count: {{ $value | printf "%.3f" }}
{{- range $x := $relates.R1}}
Loki log time: {{(nanoTime $x.Fields.__time__ 8).Format "2006-01-02T15:04:05Z07:00"}}
Loki Log line: {{$x.Fields.__log__}}
{{- end}}
{{- end}}

Loki

Core concepts#

1. Threshold mode#

Configuration#

How it works#

Recovery logic#

2. Data exists mode#

Configuration#

Pros and cons#

Recovery logic#

3. No data mode#

Configuration#

Typical applications#

4. Get original logs when alerting#

Core concepts

1. Threshold mode

Configuration

How it works

Recovery logic

2. Data exists mode

Configuration

Pros and cons

Recovery logic

3. No data mode

Configuration

Typical applications

4. Get original logs when alerting