10.18. Monitoring Nodes and Virtual Environments via Prometheus

Note

The collected statistics are only intended for monitoring and are not suitable for billing purposes.

You can monitor nodes running Virtuozzo Hybrid Server 7.5 and newer as well as VEs hosted on them via Prometheus. A typical list of required components includes:

  • Prometheus, a service that grabs statistics from exporters and stores it in a time series database.
  • Alertmanager, a service that receives alerts from Prometheus and handles their delivery via various communication channels.
  • Grafana, a service that provides a web panel with flexible dashboards and supports Prometheus as a data source.
  • Exporters, services that are installed on Virtuozzo Hybrid Server nodes and export metrics via a simple HTTP server.

This guide describes how to install the exporters, configure an existing Prometheus service, and import the dashboards to an existing Grafana panel. For details on installing Prometheus, Alertmanager, and Grafana, see the respective documentation:

10.18.1. Installing the Exporters

Perform these steps on each Virtuozzo Hybrid Server 7.5 node that you want to monitor.

  1. Install exporter packages:

    # yum install node_exporter libvirt_exporter
    
  2. Configure the firewall

    # firewall-cmd --permanent --zone=public --add-rich-rule='
        rule family="ipv4"
        source address="<prom_IP>/32"
        port protocol="tcp" port="9177" accept'
    # firewall-cmd --permanent --zone=public --add-rich-rule='
        rule family="ipv4"
        source address="<prom_IP>/32"
        port protocol="tcp" port="9100" accept'
    # firewall-cmd --reload
    

    Where <prom_IP> is the Prometheus IP address, port 9177 is used by the libvirt exporter, and port 9100 is used by the node exporter.

    It is recommended to expose the metrics only to the Prometheus server. Unrestricted access to the metrics can be a security and stability risk.

  3. Launch the exporters:

    # systemctl start node_exporter
    # systemctl start libvirt-exporter
    

After setting up the exporters, on any Virtuozzo Hybrid Server 7.5 node, obtain the sample configuration, rules, and alerts for Prometheus and dashboards for Grafana:

# yum install vz-prometheus-cfg

The files will be placed in /usr/share/vz-prometheus-cfg/. For example:

# tree /usr/share/vz-prometheus-cfg/
/usr/share/vz-prometheus-cfg/
├── alerts
│   ├── vstorage-alerts.yml
│   └── vz-alerts.yml
├── dashboards
│   ├── grafana_hn_dashboard.json
│   ├── grafana_ve_dashboard.json
│   ├── grafana_win_ct_hn_dashboard.json
│   └── grafana_win_ct_ve_dashboard.json
├── prometheus-example.yml
├── rules
│   ├── vstorage-rules.yml
│   ├── vz-rules.yml
│   └── win_ct-rules.yml
└── targets
    ├── targets-example.yml
    └── vstorage-targets-example.yml

10.18.2. Configuring Prometheus

You will need to configure Prometheus so it can start collecting metrics from Virtuozzo Hybrid Server nodes. To do this, modify prometheus.yml based on the sample prometheus-example.yml shipped with vz-prometheus-cfg.

  1. Copy the rule and alert files to the Prometheus server and set their paths in rule_files in prometheus.yml (see the example further).

  2. Create target files that contain information about exporters you want to scrape. By using multiple target files you can group nodes by attributes like datacenter, cluster, and such. The following examples create a server group cluster1 populated with five nodes and scrape their node and libvirt exporters, respectively:

    # cat cluster1-node.yml
    - labels:
        group: cluster1
      targets:
      - hn01.cluster1.tld:9100
      - hn02.cluster1.tld:9100
      - hn03.cluster1.tld:9100
      - hn04.cluster1.tld:9100
      - hn05.cluster1.tld:9100
    
    # cat cluster1-libvirt.yml
    - labels:
        group: cluster1
      targets:
      - hn01.cluster1.tld:9177
      - hn02.cluster1.tld:9177
      - hn03.cluster1.tld:9177
      - hn04.cluster1.tld:9177
      - hn05.cluster1.tld:9177
    
  3. Set paths to target files in scrape_configs in prometheus.yml (see the example further).

A final Prometheus configuration file may look like this:

# cat prometheus.yml
global:
  scrape_interval:     1m
  evaluation_interval: 1m
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - localhost:9093
rule_files:
  - /prometheus-2.21.0.linux-amd64/rules/vz-rules.yml
  - /prometheus-2.21.0.linux-amd64/rules/vstorage-rules.yml
  - /prometheus-2.21.0.linux-amd64/alerts/vz-alerts.yml
  - /prometheus-2.21.0.linux-amd64/alerts/vstorage-alerts.yml
scrape_configs:
  - job_name: 'prometheus'
    static_configs:
    - targets: ['localhost:9090']
  - job_name: node
    relabel_configs:
    - source_labels: [__address__]
      target_label: instance
      regex: (.*)[:].+
    file_sd_configs:
      - files:
        - /prometheus-2.21.0.linux-amd64/targets/cluster1-node.yml
  - job_name: libvirt
    relabel_configs:
    - source_labels: [__address__]
      target_label: instance
      regex: (.*)[:].+
    file_sd_configs:
      - files:
        - /prometheus-2.21.0.linux-amd64/targets/cluster1-libvirt.yml

After editing the Prometheus configuration file, restart the prometheus and alertmanager services.

10.18.3. Configuring Grafana

To see the data collected from the nodes in Grafana, do the following in the Grafana web panel:

  1. If you have not already done so, add the configured Prometheus server as a data source:
    1. Navigate to Configuration -> Data Sources.
    2. Click Add data source, select Prometheus..
    3. Enter a name for the data source.
    4. Specify the Prometheus IP address and port.
    5. Click Save & Test.
  2. Import Virtuozzo Hybrid Server dashboards. Perform these steps for each JSON file shipped with vz-prometheus-cfg.
    1. Navigate to Dashboards -> Manage.
    2. Click Import and Upload JSON file. Select a JSON file with a Grafana dashboard.
    3. Select the previously configured Prometheus data source.
    4. Click Import.

The Virtuozzo Hybrid Server dashboards are now available in Dashboards Home.