Alertmanager

This post describes the setup of a dead man’s switch for Prometheus / Alertmanager in a kubernetes cluster. A primary goal of the monitoring and alerting system is to generate alerts as soon as possible when problems occur, so administrators can react promptly and the impact on users can be limited. But what happens, if the monitoring system itself is impaired? In case of an outage of the kubernetes cluster where Prometheus is installed, it is highly probable that no alerts are generated....

Prometheus - Alertmanager routing with AlertmanagerConfig This post will describe the routing for Alertmanager with the declarative AlertmanagerConfig resource provided by the Prometheus-Operator in Kubernetes. Scenario Our alerts in the Kubernetes cluster are forwarded to Mattermost via the Alertmanager. This is the scenario i will cover in the post: All alerts go to a main Mattermost channel (aws-alerts) Each customer deployment is in a separate Kubernetes namespace Each customer can have multiple modules represented by a Kubernetes deployment for each module....