High Cardinality Labels Are Killing Your Prometheus: How to Find and Fix Them
In Prometheus, memory usage is dominated by the number of time series currently stored in the head block (plus indexes). Your goal is to find which metrics/labels/targets create the most series and which ones churn (get created/removed frequently). Use the built-in TSDB status endpoint Prometheus exposes a helper endpoint that lists the top offenders: http://<prometheus>:9090/api/v1/status/tsdb?limit=50 Useful sections in the JSON: seriesCountByMetricName – metric names with the most series labelValueCountByLabelName – label names with the most distinct values seriesCountByLabelPair – specific label=value pairs that explode series (Recent versions) memoryInBytesByLabelName – memory cost per label name You can filter further with match[], for example: ...