Prometheus Alertmanager 설치

• github URL : https://github.com/prometheus/alertmanager
• Prometheus에서 발생한 알람을 Alertmanager에 전달

Alertmanager 설치

1. Alertmanager 다운로드

# Alertmanager 다운로드
$ cd /root
$ wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
$ tar xvzf alertmanager-0.24.0.linux-amd64.tar.gz
alertmanager-0.24.0.linux-amd64/
alertmanager-0.24.0.linux-amd64/alertmanager.yml
alertmanager-0.24.0.linux-amd64/LICENSE
alertmanager-0.24.0.linux-amd64/NOTICE
alertmanager-0.24.0.linux-amd64/alertmanager
alertmanager-0.24.0.linux-amd64/amtool

$ mkdir /etc/alertmanager
$ cp /root/alertmanager-0.24.0.linux-amd64/* /etc/alertmanager/
$ rm -rf /root/alertmanager-0.24.0.linux-amd64

2. systemd로 Alertmanager 서비스 관리

# Alertmanager 서비스를 systemd로 관리
$ cat << EOF | tee /usr/lib/systemd/system/alertmanager.service
[Unit]
Description=Alert Manager
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/etc/alertmanager/alertmanager --config.file=/etc/alertmanager/alertmanager.yml

[Install]
WantedBy=multi-user.target
EOF

# Alertmanager 서비스 실행  
$ /bin/systemctl daemon-reload
$ /bin/systemctl restart alertmanager.service
$ /bin/systemctl status alertmanager.service
Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-04 12:38:19 KST; 5s ago
 Main PID: 55991 (alertmanager)
    Tasks: 24
   Memory: 28.0M
   CGroup: /system.slice/alertmanager.service
           └─55991 /etc/alertmanager/alertmanager --config.file=/etc/alertmanager/alertmanager.yml

Alertmanger의 설정 파일인 alertmanager.yml 생성

• Slack App의 웹훅 URL을 설정

  $ vi /etc/alertmanager/alertmanager.yml
  global:
    resolve_timeout: 5m
    slack_api_url: "https://hooks.slack.com/services/???????????????????????????"    # slack 웹훅 URL 필요
  
  route:
    receiver: 'slack-notifications'
    repeat_interval: 2m
  receivers:
    - name: 'slack-notifications'
      slack_configs:
      - channel: '[#slack channel 이름]'
        username: "Prometheus"
        send_resolved: true
        title: "{{ range .Alerts }}{{ .Annotations.summary }}\n{{ end }}"
        text: "{{ range .Alerts }}{{ .Annotations.description }}\n{{ end }}"
  
  # alertmanager 재실행 및 상태 확인
  $ systemctl restart alertmanager
  $ systemctl status alertmanager
  ● alertmanager.service - Alert Manager
     Loaded: loaded (/usr/lib/systemd/system/alertmanager.service; disabled; vendor preset: disabled)
     Active: active (running) since Tue 2022-04-05 01:56:21 KST; 4s ago
   Main PID: 108698 (alertmanager)
     Tasks: 24
   Memory: 26.6M
   CGroup: /system.slice/alertmanager.service
            └─108698 /etc/alertmanager/alertmanager --config.file=/etc/alertmanager/alertmanager.yml

Alertmanger의 설정을 prometheus.yml 파일에 추가

$ vi /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prometheus
    replica: 0

rule_files:
  - '/etc/prometheus/rules/hardware_rule.yml'

alerting:
  alertmanagers:
  - scheme: http
    static_configs:
    - targets:
      - "[alertmanager가 설치된 서버 IP]:9093"   # <alertmanager ip:port>

scrape_configs:
  - job_name: 'node-exporter'
    scrape_interval: 5s
    static_configs:
      - targets:
        - [node-exporter 서버 IP]:9100

  - job_name: 'dellhw_exporter'
    scrape_interval: 60s
    static_configs:
      - targets:
        - [dellhw_exporter 서버 IP]:9137
        - [dellhw_exporter 서버 IP]:9137

Alertmanager 웹 UI

• http://[alertmanager가 설치된 서버 IP:9093/#/alerts)에서 Prometheus에서 발생한 알람
• 알람 관리 UI 확인