Donate to support Ukraine's independence.
Generated by AI

High Cardinality Labels Are Killing Your Prometheus: How to Find and Fix Them

In Prometheus, memory usage is dominated by the number of time series currently stored in the head block (plus indexes). Your goal is to find which metrics/labels/targets create the most series and which ones churn (get created/removed frequently). Use the built-in TSDB status endpoint Prometheus exposes a helper endpoint that lists the top offenders: http://<prometheus>:9090/api/v1/status/tsdb?limit=50 Useful sections in the JSON: seriesCountByMetricName – metric names with the most series labelValueCountByLabelName – label names with the most distinct values seriesCountByLabelPair – specific label=value pairs that explode series (Recent versions) memoryInBytesByLabelName – memory cost per label name You can filter further with match[], for example: ...

September 14, 2025 · 3 min · 445 words · Serhii Kaidalov
Generated by AI

How to Test Alertmanager Routing Without a Real Incident

When configuring new receivers or testing templates in Alertmanager, you might need to generate dummy alerts to see how everything works. Instead of manually creating and sending alerts, I use a simple bash script to streamline the process. The Script Below is the script I use to generate and resolve dummy alerts. It sends a POST request to Alertmanager’s API to simulate an alert. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 #!/bin/bash name=$RANDOM url='http://localhost:9093/api/v2/alerts' default_severity='warning' instance='127.0.0.1' instance_name='test-instance' send_alert() { local status=$1 local custom_severity=$2 local current_severity=${custom_severity:-$default_severity} curl -XPOST $url -H "Content-Type: application/json" -d "[ { \"status\": \"$status\", \"labels\": { \"alertname\": \"$name\", \"service\": \"my-service\", \"severity\":\"$current_severity\", \"instance\": \"$instance\", \"instance_name\": \"$instance_name\" }, \"annotations\": { \"summary\": \"This is a test alert\", \"title\": \"Test Alert\", \"description\": \"Test alert has been triggered.\" }, \"generatorURL\": \"https://prometheus.local/<generating_expression>\" } ]" echo "" } # Main script echo "Firing up alert $name" send_alert "firing" "$1" read -p "Press enter to resolve alert" echo "Sending resolve" send_alert "resolved" "$1" How It Works The script starts by generating a random alert name ($RANDOM) to ensure each alert is unique. It sets a default severity (warning) and other labels such as instance and instance_name. The send_alert function sends a POST request to Alertmanager’s API endpoint (http://localhost:9093/api/v2/alerts) with a payload containing the alert’s status, labels, and annotations. The script fires an alert, waits for user input, and then resolves the alert. Example Output Running the script will output something like this: ...

January 1, 2025 · 3 min · 463 words · Serhii Kaidalov