Install Prometheus Alertmanager on VPS

September 2026 · 9 min read

Prometheus Alertmanager handles alerts sent by Prometheus, providing grouping, deduplication, and routing to receivers like Email, Slack, LINE Notify, PagerDuty, or any custom webhook. Many sysadmins set up Prometheus and Grafana but skip Alertmanager — meaning no notifications reach them when a server goes down. This guide walks through a complete setup from installation to receiving real alerts in Slack.

Prerequisites

• VPS running Ubuntu 22.04 / Debian 12 or later
• Prometheus already installed (see Grafana + Prometheus guide)
• Node Exporter running on the VPS
• Slack Workspace or Email account to receive alerts

Download and Install Alertmanager

Download the latest binary directly from GitHub releases — no package manager needed:

# Download Alertmanager
cd /tmp
curl -LO https://github.com/prometheus/alertmanager/releases/download/v0.27.0/alertmanager-0.27.0.linux-amd64.tar.gz

# Extract archive
tar xzf alertmanager-0.27.0.linux-amd64.tar.gz

# Move binaries to /usr/local/bin
sudo mv alertmanager-0.27.0.linux-amd64/alertmanager /usr/local/bin/
sudo mv alertmanager-0.27.0.linux-amd64/amtool /usr/local/bin/

# Verify installation
alertmanager --version

Create System User and Directories

# Create dedicated system user
sudo useradd --no-create-home --shell /bin/false alertmanager

# Create config and data directories
sudo mkdir -p /etc/alertmanager /var/lib/alertmanager

# Set proper ownership
sudo chown alertmanager:alertmanager /etc/alertmanager /var/lib/alertmanager
sudo chown alertmanager:alertmanager /usr/local/bin/alertmanager /usr/local/bin/amtool

Configure alertmanager.yml

The main config file controls all routing and receivers. This example routes critical alerts to Slack and warnings to Email:

sudo nano /etc/alertmanager/alertmanager.yml

global:
  resolve_timeout: 5m
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[email protected]'
  smtp_auth_username: '[email protected]'
  smtp_auth_password: 'your-app-password'
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'

route:
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack-default'
  routes:
    - match:
        severity: critical
      receiver: 'slack-critical'
      repeat_interval: 1h
    - match:
        severity: warning
      receiver: 'email-warning'

receivers:
  - name: 'slack-default'
    slack_configs:
      - channel: '#alerts'
        title: '{{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        send_resolved: true

  - name: 'slack-critical'
    slack_configs:
      - channel: '#alerts-critical'
        title: '🔴 CRITICAL: {{ .GroupLabels.alertname }}'
        text: |
          *Instance:* {{ .GroupLabels.instance }}
          {{ range .Alerts }}*Description:* {{ .Annotations.description }}{{ end }}
        send_resolved: true
        color: '{{ if eq .Status "firing" }}danger{{ else }}good{{ end }}'

  - name: 'email-warning'
    email_configs:
      - to: '[email protected]'
        subject: '[Warning] {{ .GroupLabels.alertname }}'
        body: |
          Alert: {{ .GroupLabels.alertname }}
          {{ range .Alerts }}
          Instance: {{ .Labels.instance }}
          Description: {{ .Annotations.description }}
          {{ end }}

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

Create Prometheus Alert Rules

Alertmanager only receives alerts from Prometheus, so you must define rules in Prometheus first:

sudo nano /etc/prometheus/rules/node_alerts.yml

groups:
  - name: node_alerts
    interval: 30s
    rules:
      # CPU above 85% for 5 minutes
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage high on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}"

      # Available RAM below 10%
      - alert: LowMemory
        expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100 < 10
        for: 2m
        labels:
          severity: critical
        annotations:
          summary: "Low memory on {{ $labels.instance }}"
          description: "Available memory is {{ $value | printf \"%.1f\" }}% on {{ $labels.instance }}"

      # Disk space below 15%
      - alert: LowDiskSpace
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 15
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is {{ $value | printf \"%.1f\" }}% available on {{ $labels.instance }}"

      # Instance unreachable
      - alert: InstanceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Instance {{ $labels.instance }} down"
          description: "{{ $labels.instance }} has been down for more than 1 minute"

Connect Prometheus to Alertmanager

Edit prometheus.yml to reference Alertmanager and load the rules files:

sudo nano /etc/prometheus/prometheus.yml

global:
  scrape_interval: 15s
  evaluation_interval: 15s

alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - localhost:9093

rule_files:
  - "/etc/prometheus/rules/*.yml"

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']

Create systemd Service

sudo nano /etc/systemd/system/alertmanager.service

[Unit]
Description=Prometheus Alertmanager
After=network.target

[Service]
User=alertmanager
Group=alertmanager
Type=simple
ExecStart=/usr/local/bin/alertmanager \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --storage.path=/var/lib/alertmanager \
  --web.listen-address=:9093 \
  --web.external-url=http://localhost:9093
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable --now alertmanager

# Reload Prometheus to pick up new config
sudo systemctl reload prometheus

Test Alert Delivery

# Send a test alert using amtool
amtool alert add alertname="TestAlert" severity="warning" \
  --alertmanager.url=http://localhost:9093 \
  instance="vps-test" \
  --annotation=summary="Test alert from amtool" \
  --annotation=description="This is a test alert"

# List active alerts
amtool alert --alertmanager.url=http://localhost:9093

# List configured silences
amtool silence --alertmanager.url=http://localhost:9093

Verify Alertmanager Is Working Correctly

# Check service status
sudo systemctl status alertmanager

# Confirm port 9093 is listening
ss -tlnp | grep 9093

# Validate config syntax
amtool check-config /etc/alertmanager/alertmanager.yml

# Follow logs in real time
sudo journalctl -u alertmanager -f

# Check health endpoint
curl -s http://localhost:9093/-/healthy

Troubleshooting Common Issues

Error: "dial tcp: connection refused" on port 9093

Alertmanager is not running, or a firewall is blocking the port. Check the service and open the port if needed.

# Check service status
sudo systemctl status alertmanager

# View error logs
sudo journalctl -u alertmanager --no-pager -n 50

# Do not expose Alertmanager directly to the internet — use a reverse proxy like Nginx

Error: "yaml: line X: did not find expected key"

The YAML in alertmanager.yml has incorrect indentation. YAML requires consistent spaces — tabs are not allowed.

# Validate config before restarting
amtool check-config /etc/alertmanager/alertmanager.yml

# Check with Python
python3 -c "import yaml; yaml.safe_load(open('/etc/alertmanager/alertmanager.yml'))"

Alert fires in Prometheus but no Slack/Email received

This can be caused by an expired Slack Webhook URL, wrong SMTP credentials, or routing config that doesn't match the alert's labels.

# View alerts received by Alertmanager
curl -s http://localhost:9093/api/v2/alerts | python3 -m json.tool

# Test which receiver an alert routes to
amtool config routes test --alertmanager.url=http://localhost:9093 \
  alertname="HighCPUUsage" severity="warning"

# Check logs for notification failures
sudo journalctl -u alertmanager | grep -i "notify\|error\|fail"

Alert keeps firing every 5 minutes even after fixing the issue

Either repeat_interval is too short, or the metric hasn't actually returned to normal. Verify the metric value in Prometheus.

# Check if the alert is still in firing state
curl -s http://localhost:9090/api/v1/alerts | python3 -m json.tool | grep '"state"'

# Create a temporary 2-hour silence while investigating
amtool silence add alertname="HighCPUUsage" \
  --alertmanager.url=http://localhost:9093 \
  --duration=2h \
  --comment="Investigating issue"

Error: "context deadline exceeded" when sending Email

The SMTP server is unresponsive, or the port is blocked by a firewall. Gmail requires an App Password, not your regular account password.

# Test SMTP connectivity
openssl s_client -starttls smtp -connect smtp.gmail.com:587

# Create a Gmail App Password at:
# https://myaccount.google.com/apppasswords
# (requires 2-Step Verification to be enabled)

Frequently Asked Questions

How does Alertmanager differ from Grafana Alerting?

Prometheus evaluates alert rules and sends fired alerts to Alertmanager, which handles routing, grouping, deduplication, and silencing. Grafana Alerting is a separate system that can query multiple data sources and has a user-friendly UI for managing alerts. For large production environments, Alertmanager is preferred because of its fine-grained inhibition and silence rules. Grafana Alerting suits teams that need multi-datasource alerting without running a separate service.

What repeat_interval should I configure?

It depends on alert severity. Critical alerts that need immediate action often use 1h so responders are reminded frequently. Non-urgent warnings may use 12h or 24h to avoid notification fatigue. Most teams start with 4h for warnings and 1h for critical alerts, then tune based on real-world experience.

How do I mute alerts during scheduled maintenance?

Use the Alertmanager Web UI or amtool to create a Silence that matches specific labels for a defined time period. Example: amtool silence add instance="vps01" --duration=2h --comment="Scheduled maintenance". Any alert matching those labels will not be sent to receivers during the silence window.