deploy/docs/operator-manual/notifications/services/alertmanager.md

Alertmanager

Parameters

The notification service is used to push events to Alertmanager, and the following settings need to be specified:

• targets - the alertmanager service address, array type
• scheme - optional, default is "http", e.g. http or https
• apiPath - optional, default is "/api/v2/alerts"
• insecureSkipVerify - optional, default is "false", when scheme is https whether to skip the verification of ca
• basicAuth - optional, server auth
• bearerToken - optional, server auth
• timeout - optional, the timeout in seconds used when sending alerts, default is "3 seconds"
• maxIdleConns - optional, maximum number of idle (keep-alive) connections across all hosts.
• maxIdleConnsPerHost - optional, maximum number of idle (keep-alive) connections per host.
• maxConnsPerHost - optional, maximum total connections per host.
• idleConnTimeout - optional, maximum amount of time an idle (keep-alive) connection will remain open before closing.

basicAuth or bearerToken is used for authentication, you can choose one. If the two are set at the same time, basicAuth takes precedence over bearerToken.

Example

Prometheus Alertmanager config

global:
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'default'
receivers:
- name: 'default'
  webhook_configs:
  - send_resolved: false
    url: 'http://10.5.39.39:10080/api/alerts/webhook'

You should turn off "send_resolved" or you will receive unnecessary recovery notifications after "resolve_timeout".

Send one alertmanager without auth

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-notifications-cm
data:
  service.alertmanager: |
    targets:
    - 10.5.39.39:9093

Send alertmanager cluster with custom api path

If your alertmanager has changed the default api, you can customize "apiPath".

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-notifications-cm
data:
  service.alertmanager: |
    targets:
    - 10.5.39.39:443
    scheme: https
    apiPath: /api/events
    insecureSkipVerify: true

Send high availability alertmanager with auth

Store auth token in cd-notifications-secret Secret and use configure in cd-notifications-cm ConfigMap.

apiVersion: v1
kind: Secret
metadata:
  name: <secret-name>
stringData:
  alertmanager-username: <username>
  alertmanager-password: <password>
  alertmanager-bearer-token: <token>

• with basicAuth

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-notifications-cm
data:
  service.alertmanager: |
    targets:
    - 10.5.39.39:19093
    - 10.5.39.39:29093
    - 10.5.39.39:39093
    scheme: https
    apiPath: /api/v2/alerts
    insecureSkipVerify: true
    basicAuth:
      username: $alertmanager-username
      password: $alertmanager-password   

• with bearerToken

apiVersion: v1
kind: ConfigMap
metadata:
  name: cd-notifications-cm
data:
  service.alertmanager: |
    targets:
    - 10.5.39.39:19093
    - 10.5.39.39:29093
    - 10.5.39.39:39093
    scheme: https
    apiPath: /api/v2/alerts
    insecureSkipVerify: true
    bearerToken: $alertmanager-bearer-token

Templates

• labels - at least one label pair required, implement different notification strategies according to alertmanager routing
• annotations - optional, specifies a set of information labels, which can be used to store longer additional information, but only for display
• generatorURL - optional, default is '{{.app.spec.source.repoURL}}', backlink used to identify the entity that caused this alert in the client

the label or annotations or generatorURL values can be templated.

context: |
  cdUrl: https://example.com/cd

template.app-deployed: |
  message: Application {{.app.metadata.name}} has been healthy.
  alertmanager:
    labels:
      fault_priority: "P5"
      event_bucket: "deploy"
      event_status: "succeed"
      recipient: "{{.recipient}}"
    annotations:
      application: '<a href="{{.context.cdUrl}}/applications/{{.app.metadata.name}}">{{.app.metadata.name}}</a>'
      author: "{{(call .repo.GetCommitMetadata .app.status.sync.revision).Author}}"
      message: "{{(call .repo.GetCommitMetadata .app.status.sync.revision).Message}}"

You can do targeted push on Alertmanager according to labels.

template.app-deployed: |
  message: Application {{.app.metadata.name}} has been healthy.
  alertmanager:
    labels:
      alertname: app-deployed
      fault_priority: "P5"
      event_bucket: "deploy"

There is a special label alertname. If you don’t set its value, it will be equal to the template name by default.