Using Filebeat for log forwarding

Cluster logs cannot be stored on nodes for a long period of time due to log rotation and limited storage space. Their retention period may be insufficient for troubleshooting purposes. You can use the built-in Filebeat service to forward log data to a centralized log management system. Filebeat monitors the predefined log files on all cluster nodes or a particular node, collects log events, and then sends them to a specific destination. By default, Filebeat is configured to work with Elasticsearch. You can, however, create a custom configuration for Filebeat to work with other log management systems, such as Logstash, Kafka, or Redis.

The Filebeat service is disabled by default. To start using it, you need to enable it first.

To configure Filebeat for Elasticsearch

Use the following command:

vinfra cluster filebeat config set --elasticsearch [--host <host>] [--port <port>] [--username <username>]
                                   [--password <password>] [--ca_cert <ca_cert>] [--cert <cert>]
                                   [--key <key>] [--nodes <nodes> | --all]
--elasticsearch
Set options for the predefined Filebeat configuration (Elasticsearch template)
--host <host>
Elasticsearch hostname or IP address
--port <port>
Elasticsearch port (default is 9200)
--username <username>
Elasticsearch username
--password <password>
Elasticsearch password
--ca_cert <ca_cert>
Path to CA certificate
--cert <cert>
Path to client certificate
--key <key>
Path to certificate key
--nodes <nodes>
A comma-separated list of node IDs or hostnames
--all
Apply the configuration on all cluster nodes

For example, to configure Filebeat to forward logs from all cluster nodes to the Elasticsearch server with the IP address 10.10.10.10, run:

# vinfra cluster filebeat config set --all --elasticsearch --host 10.10.10.10 --username root --password password

To configure Filebeat for other destinations

  1. Create a custom Filebeat configuration file. For example, to configure Filebeat to forward the audit log from the management node node001 to the Logstash server with the IP address 10.10.10.10, update the /etc/filebeat/filebeat.yml file as follows:

    # cat > /etc/filebeat/filebeat.yml <<\EOT 
    filebeat.inputs:
     - type: type: filestream
       id: my-filestream-id
       enabled: true
       paths:
        - /var/log/vstorage-ui-backend/audit.log
       fields:
        log_type: audit_log
       fields_under_root: true 
    output.logstash:
      hosts: ["10.10.10.10:5044"]
    EOT

    For more details on Filebeat options, refer to the official documentation.

  2. Use the created file to update the Filebeat configuration:

    vinfra cluster filebeat config set --filename <filename> [--nodes <nodes> | --all]
    --filename <filename>
    Path to the Filebeat configuration file to upload
    --nodes <nodes>
    A comma-separated list of node IDs or hostnames
    --all
    Apply the configuration on all cluster nodes

    For example, to configure Filebeat to forward the specified logs from the management node node001, run:

    # vinfra cluster filebeat config set --nodes node001 --filename filebeat.yml

To enable and start Filebeat

  1. Enable the service:

    # vinfra cluster filebeat enable [--nodes <nodes> | --all]
  2. Start Filebeat:

    # vinfra cluster filebeat start [--nodes <nodes> | --all]

To update the Filebeat configuration

  1. Reload the Filebeat configuration:

    # vinfra cluster filebeat config reload [--nodes <nodes> | --all]
  2. Restart the service:

    # vinfra cluster filebeat restart [--nodes <nodes> | --all]

To stop and disable Filebeat

  1. Stop Filebeat:

    # vinfra cluster filebeat stop [--nodes <nodes> | --all]
  2. Disable the service:

    # vinfra cluster filebeat disable [--nodes <nodes> | --all]

To delete the Filebeat configuration

Use the following command:

vinfra cluster filebeat config delete [--nodes <nodes> | --all]
--nodes <nodes>
A comma-separated list of node IDs or hostnames
--all
Delete the Filebeat configuration from all cluster nodes

For example, to delete the Filebeat configuration from the node node003, run:

# vinfra cluster filebeat config delete --nodes node003