Using Filebeat for log forwarding

Cluster logs cannot be stored on nodes for a long period of time due to log rotation and limited storage space. Their retention period may be insufficient for troubleshooting purposes. You can use the built-in Filebeat service to forward log data to a centralized log management system. Filebeat monitors the predefined log files on all cluster nodes or a particular node, collects log events, and then sends them to a specific destination. By default, Filebeat is configured to work with Elasticsearch. You can, however, create a custom configuration for Filebeat to work with other log management systems, such as Logstash, Kafka, or Redis.

The Filebeat service is disabled by default. To start using it, you need to enable it first.

To configure Filebeat for Elasticsearch

Use the following command:

vinfra cluster filebeat config set --elasticsearch [--host <host>] [--port <port>] [--username <username>]
                                   [--password <password>] [--ca_cert <ca_cert>] [--cert <cert>]
                                   [--key <key>] [--nodes <nodes> | --all]

--elasticsearch: Set options for the predefined Filebeat configuration (Elasticsearch template)
--host <host>: Elasticsearch hostname or IP address
--port <port>: Elasticsearch port (default is 9200)
--username <username>: Elasticsearch username
--password <password>: Elasticsearch password
--ca_cert <ca_cert>: Path to CA certificate
--cert <cert>: Path to client certificate
--key <key>: Path to certificate key
--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Apply the configuration on all cluster nodes

For example, to configure Filebeat to forward logs from all cluster nodes to the Elasticsearch server with the IP address 10.10.10.10, run:

# vinfra cluster filebeat config set --all --elasticsearch --host 10.10.10.10 --username root --password password

To configure Filebeat for other destinations

Create a custom Filebeat configuration file. For example, to configure Filebeat to forward the audit log from the management node node001 to the Logstash server with the IP address 10.10.10.10, update the /etc/filebeat/filebeat.yml file as follows:
```
# cat > /etc/filebeat/filebeat.yml <<\EOT 
filebeat.inputs:
 - type: type: filestream
   id: my-filestream-id
   enabled: true
   paths:
    - /var/log/vstorage-ui-backend/audit.log
   fields:
    log_type: audit_log
   fields_under_root: true 
output.logstash:
  hosts: ["10.10.10.10:5044"]
EOT
```
For more details on Filebeat options, refer to the official documentation.
Use the created file to update the Filebeat configuration:
```
vinfra cluster filebeat config set --filename <filename> [--nodes <nodes> | --all]
```
--filename <filename>

Path to the Filebeat configuration file to upload

--nodes <nodes>

A comma-separated list of node IDs or hostnames

--all

Apply the configuration on all cluster nodes

For example, to configure Filebeat to forward the specified logs from the management node node001, run:
```
# vinfra cluster filebeat config set --nodes node001 --filename /etc/filebeat/filebeat.yml
```

To enable the Filebeat service

Use the following command:

vinfra cluster filebeat enable [--nodes <nodes> | --all]

--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Enable Filebeat on all cluster nodes

For example, to enable the Filebeat service on all cluster nodes, run:

# vinfra cluster filebeat enable --all

To update the Filebeat configuration

Use the following command:

vinfra cluster filebeat config reload [--nodes <nodes> | --all]

--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Reload the Filebeat configuration on all cluster nodes

For example, to reload the Filebeat configuration on all cluster nodes, run:

# vinfra cluster filebeat config reload --all

To disable the Filebeat service

Use the following command:

vinfra cluster filebeat disable [--nodes <nodes> | --all]

--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Disable Filebeat on all cluster nodes

For example, to disable the Filebeat service on all cluster nodes, run:

# vinfra cluster filebeat disable --all

To delete the Filebeat configuration

Use the following command:

vinfra cluster filebeat config delete [--nodes <nodes> | --all]

--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Delete the Filebeat configuration from all cluster nodes

For example, to delete the Filebeat configuration from the node node003, run:

# vinfra cluster filebeat config delete --nodes node003