Download all required packages

Go to the download page, find these packages as below,

https://prometheus.io/download/

  1. Prometheus

    https://github.com/prometheus/prometheus/releases/download/v2.37.5/prometheus-2.37.5.linux-amd64.tar.gz

  2. Alertmanager

    https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz

  3. Node_exporter

    https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz

Allow 9090 port open

ufw allow 9090/tcp
ufw reload

Configure and start

tar xvfz prometheus-2.37.5.linux-amd64.tar.gz
cd prometheus-2.37.5.linux-amd64/
mv prometheus-2.37.5.linux-amd64 /usr/local/prometheus
useradd -M -s /sbin/nologin prometheus 
cd /usr/local/prometheus
mkdir data
chown -R prometheus:prometheus /usr/local/prometheus

Add to system service

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
WorkingDirectory=/usr/local/prometheus
ExecStart=/usr/local/prometheus/prometheus

[Install]
WantedBy=multi-user.target

Save the file to /usr/lib/systemd/system/prometheus.service

Reload system service

systemctl daemon-reload

Start prometheus

systemctl start prometheus
systemctl enable prometheus

Upload the node exporter to all servers

wget https://github.com/prometheus/node_exporter/releases/download/v1.5.0/node_exporter-1.5.0.linux-amd64.tar.gz
tar xvf node_exporter-1.5.0.linux-amd64.tar.gz
mv node_exporter-1.5.0.linux-amd64.tar.gz /usr/local/node_exporter
useradd -M -s /sbin/nologin prometheus
chown -R prometheus:prometheus /usr/local/node_exporter
ufw allow 9100/tcp

Add to service

[Unit]
Description=node_exporter
After=network.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecStart=/usr/local/node_exporter/node_exporter
  --web.listen-address=:9100
  --web.telemetry-path=/metrics
  --log.level=info
  --log.format=logfmt
Restart=always

[Install]
WantedBy=multi-user.target

Reload system service

systemctl daemon-reload

Start node_exporter

systemctl start node_exporter
systemctl enable node_exporter

get the local data

curl http://localhost:9100/metrics | head

But we won’t use this method to get the data, we will get the data through the public ip. Let’s go back to the Prometheus server.

vim /usr/local/prometheus/prometheus.yml

  - job_name: "node"
    static_configs:
      - targets: ['remote_ip:9100']

Add a new job name into the scrape_configs.

systemctl restart prometheus
cd /usr/local/prometheus/
rm -rf data # if somedata is missing
systemctl restart prometheus

Install Grafana

Go to the webpage https://grafana.com/grafana/download

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/enterprise/release/grafana-enterprise_9.3.2_amd64.deb
sudo dpkg -i grafana-enterprise_9.3.2_amd64.deb
sudo /bin/systemctl daemon-reload
sudo /bin/systemctl enable grafana-server
### You can start grafana-server by executing
sudo /bin/systemctl start grafana-server

ufw allow 3000/tcp

Then you can open the grafana server with the port http://ip:3000

Alertmanager

wget https://github.com/prometheus/alertmanager/releases/download/v0.24.0/alertmanager-0.24.0.linux-amd64.tar.gz
tar xvf alertmanager-0.24.0.linux-amd64.tar.gz
mv alertmanager-0.24.0.linux-amd64 /usr/local/alertmanager
chown -R prometheus:prometheus /usr/local/alertmanager

add alertmanager to service

[Unit]
Description=alertmanager
After=network.target

[Service]
User=prometheus
Group=prometheus
WorkingDirectory=/usr/local/prometheus/alertmanager
ExecStart=/usr/local/prometheus/alertmanager/alertmanager --log.level=debug --log.format=json

[Install]
WantedBy=multi-user.target

Save it to /usr/lib/systemd/system/alertmanager.service

systemctl daemond-reload
systemctl start alertmanager
systemctl enable alertmanager

Prometheus integrated with alertmanager

vim /usr/local/prometheus/prometheus.yml

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          - 127.0.0.1:9093  # change this to your ip

add alert rules

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - "rules/*rules.yml"

mkdir -p /usr/local/prometheus/rules

Add some information into the rule

vim /usr/local/prometheus/rules/node_rules.yml

groups:
  - name: node-alert
    rules:
      - alert: disk-full
        expr: (1-(node_filesystem_avail_bytes{mountpoint="/",fstype=~"ext4|xfs"} / node_filesystem_size_bytes {mountpoint="/",fstype=~"ext4|xfs"})) * 100 >10
        for: 1m
        labels:
          serverity: page
        annotations:
          summary: "{{ $labels.instance }} disk full"
          description: "{{ $labels.instance }} disk > {{ $value }} "

Restart the prometheus

systemctl restart prometheus

There should be some alert notification information on your website.

push alert notification to social media or email

Add Basic Auth for prometheus and node_exporter

# generate a password
htpasswd -nBC 12 '' | tr -d ':\n'

This command is from httpd-tools, if you don’t know whether the tool was been installed, you can query with this command as follow,

rpm -qa|grep httpd-tools

Create a file on node_exporter folder.

# config.yml
basic_auth_users:
  admin: $2y$12$DySdA6rOwmXt6d3R4r8UQeJ8Trvu1LtLPjkHaWeQEbND4ydwKX66x

Add this config file to your node exporter when it starts

# /usr/lib/systemd/system/node_exporter.service
--web.config=/usr/local/node_exporter/config.yml

Reload and restart the node_exporter service

systemctl daemon-reload
systemctl restart node_exporter

did the same process on prometheus.

And don’t forget to update your grafana server.