3.2 Monitoring stack configuration

Task 3.2.1: Enable alerting

Let’s take a look at an example of how you can send alerts to a receiver of your choice. Configuring a new receiver can either be done using the web console or by directly configuring the appropriate secret. Before going into detail on where this configuration lies, let’s have a look at an example:

---
apiVersion: v1
kind: Secret
metadata:
  name: alertmanager-main
  namespace: openshift-monitoring
stringData:
  alertmanager.yaml: |
    ---
    global:
      resolve_timeout: 5m
      smtp_from: monitoring@ops-training.openshift.ch
      smtp_smarthost: smtp.ops-training.openshift.ch:587
      smtp_hello: ops-training.openshift.ch
      smtp_auth_username: test
      smtp_auth_password: test
      smtp_require_tls: true
    inhibit_rules:
      - equal:
          - namespace
          - alertname
        source_match:
          severity: critical
        target_match_re:
          severity: warning|info
      - equal:
          - namespace
          - alertname
        source_match:
          severity: warning
        target_match_re:
          severity: info
    receivers:
      - name: Default
      - name: Watchdog
      - name: Critical
      - name: mail
        email_configs:
          - to: ops@ops-training.openshift.ch
    route:
      group_by:
        - namespace
      group_interval: 5m
      group_wait: 30s
      receiver: Default
      repeat_interval: 12h
      routes:
        - match:
            alertname: Watchdog
          receiver: Watchdog
        - match:
            severity: critical
          receiver: mail
type: Opaque

The main components that can be configured when applying a custom Alertmanager configuration comprise:

• receivers: With a receiver, one or more notification types such as mails, webhooks or messaging platforms like Slack or PagerDuty can be defined.
• routes: With routing blocks, a tree of routes and child routes can be defined. Each routing block has a matcher which can match one or several labels of an alert. Per block, one receiver can be specified, or if empty, the default receiver is taken.
• Watchdog: There is a default Watchdog alert that is periodically firing to make sure that the complete alerting pipeline is functional. It is best practice to implement a Deadman’s switch on the receiving side to get notified when sending alerts should fail.
• inhibit_rules: You can group alerts so if one alert of this group was triggered, others will not. This is to prevent notification flooding. As an example, imagine an etcd member is unavailable (severity: critical). As a consequence, this would also trigger a high number of failed leader proposals (severity: warning). When fixing the unavailable etcd member, both alerts will be resolved, so you only care about one of them.

You are now going to configure Alertmanager so that alerts created by Prometheus are sent to your alerts channel on Slack. In order to do this, navigate to Administration, Cluster Settings, Global Configuration, Alertmanager on the web console. Here you can see the pre-configured Receivers. Click on Configure next to the Default receiver. Now fill in the Slack API URL and Channel information provided to you by your trainer and click Save.

Switch to Slack and check if notifications are showing up in your alerts channel. This might take some minutes so don’t hesitate to continue the lab and check back later.

Configuring Alertmanager receivers via web console is easier than writing the configuration manually. However, the time will come when you want to configure receivers automatically or with more advanced settings not exposed on the web console. In order to do so, you adapt the alertmanager-main secret in the openshift-monitoring namespace. Have a look at how Alertmanager is currently configured. To do that, extract the alertmanager-main secret’s information in the openshift-monitoring namespace.

oc -n openshift-monitoring extract secrets/alertmanager-main

Task 3.2.2: Persistence and metrics retention

By default, the monitoring stack loses all scraped metrics on restarts because it does not persist its data. Let’s make our Prometheus time series database persistent and increase the metrics retention to 2 days so the volume won’t fill up.

To do so, edit the cluster-monitoring-config configmap in the openshift-monitoring namespace.

oc -n openshift-monitoring edit cm cluster-monitoring-config

Add the following snippet:

    ...
    prometheusK8s:
      ...
      retention: 2d
      volumeClaimTemplate:
       spec:
         resources:
           requests:
             storage: 5Gi
    ...

It is advisable to persist Alertmanager itself, too. This allows Alertmanager to retain active alerts on restarts and therefore prevents existing alerts from triggering again. Edit the same configmap as before (cluster-monitoring-config in openshift-monitoring).

oc -n openshift-monitoring edit cm cluster-monitoring-config

Add the following config:

    ...
    alertmanagerMain:
      ...
      volumeClaimTemplate:
       spec:
         resources:
           requests:
             storage: 1Gi
      ...

After adding the above configuration to the ConfigMap, the Cluster Monitoring Operator will create the required persistent volume claims automatically. Check for these new persistent volume claims.

oc -n openshift-monitoring get pvc

NAME                                       STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
alertmanager-main-db-alertmanager-main-0   Bound    pvc-49453e5f-1e04-4d00-88cb-cd04528c9700   1Gi        RWO            gp2            5m43s
alertmanager-main-db-alertmanager-main-1   Bound    pvc-42b8083b-5dc2-4218-a8e2-15111e0a87e7   1Gi        RWO            gp2            5m43s
alertmanager-main-db-alertmanager-main-2   Bound    pvc-824824e3-ee0b-49c6-a786-3efd86055acb   1Gi        RWO            gp2            5m42s
prometheus-k8s-db-prometheus-k8s-0         Bound    pvc-19894227-c8ad-4ac3-a793-71a4a409fd23   5Gi        RWO            gp2            6m32s
prometheus-k8s-db-prometheus-k8s-1         Bound    pvc-11fcf37e-5803-41e7-95ae-f2779010b201   5Gi        RWO            gp2            6m32s