Understanding System Health

Build an overall picture of whether Prometheus itself is healthy and whether it is successfully monitoring what it should. A good health check covers the server, its targets, and the data being collected.

Getting oriented

• healthy and ready report whether the server is up and able to serve queries.
• build_info and runtime_info identify the version, storage retention, and runtime settings; flags exposes the full command line.
• list_targets shows every scrape target and its health; list_alerts shows what is actively firing.
• tsdb_stats summarizes series counts, cardinality, and ingestion (the load side of the picture).
• query Prometheus's self-monitoring metrics directly: rate(prometheus_tsdb_head_samples_appended_total5m) shows ingestion throughput, and counters like prometheus_rule_evaluation_failures_total or prometheus_notifications_dropped_total should sit at zero -- any steady rate on them is a problem in itself.

Topics worth exploring

Treat these as starting points and follow what the data shows:

• Firing alerts: anything already alerting is the fastest pointer to known problems; the labels identify the affected services.
• Target health: down or flapping targets in list_targets mean missing data, and their lastError usually says why.
• Scrape performance: how close scrapes run to their timeout, e.g. with query: topk(10, scrape_duration_seconds)
• TSDB pressure: series counts and per-metric cardinality from tsdb_stats; sudden growth is an early warning sign.
• Prometheus's own resources: if node_exporter or cadvisor metrics are available, check CPU, memory, and disk headroom, e.g.: process_resident_memory_bytes{job="prometheus"}
• Data freshness: confirm critical metrics are current, e.g. query up, or time() - timestamp() to spot staleness.

Reporting findings

Summarize server status, alert and target health, and TSDB load, and call out anything that needs attention together with the evidence that shows it.