Using Filebeat for log forwarding

Cluster logs cannot be stored on nodes for a long period of time due to log rotation and limited storage space. Their retention period may be insufficient for troubleshooting purposes. You can use the built-in Filebeat service to forward log data to a centralized log management system. Filebeat monitors the predefined log files on all cluster nodes or a particular node, collects log events, and then sends them to a specific destination. By default, Filebeat is configured to work with Elasticsearch. You can, however, create a custom configuration for Filebeat to work with other log management systems, such as Logstash, Kafka, or Redis.

The Filebeat service is disabled by default. To start using it, you need to enable it first.

To configure Filebeat for Elasticsearch

Use the following command:

vinfra cluster filebeat config set --elasticsearch [--host <host>] [--port <port>] [--username <username>]
                                   [--password <password>] [--ca_cert <ca_cert>] [--cert <cert>]
                                   [--key <key>] [--nodes <nodes> | --all]

--elasticsearch: Set options for the predefined Filebeat configuration (Elasticsearch template)
--host <host>: Elasticsearch hostname or IP address
--port <port>: Elasticsearch port (default is 9200)
--username <username>: Elasticsearch username
--password <password>: Elasticsearch password
--ca_cert <ca_cert>: Path to CA certificate
--cert <cert>: Path to client certificate
--key <key>: Path to certificate key
--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Apply the configuration on all cluster nodes

For example, to configure Filebeat to forward logs from all cluster nodes to the Elasticsearch server with the IP address 10.10.10.10, run:

# vinfra cluster filebeat config set --all --elasticsearch --host 10.10.10.10 --username root --password password

To configure Filebeat for other destinations

Create a custom Filebeat configuration file. For example, to configure Filebeat to forward the audit log from the management node node001 to the Logstash server with the IP address 10.10.10.10, update the /etc/filebeat/filebeat.yml file as follows:
```
# cat > /etc/filebeat/filebeat.yml <<\EOT 
filebeat.inputs:
 - type: type: filestream
   id: my-filestream-id
   enabled: true
   paths:
    - /var/log/vstorage-ui-backend/audit.log
   fields:
    log_type: audit_log
   fields_under_root: true 
output.logstash:
  hosts: ["10.10.10.10:5044"]
EOT
```
For more details on Filebeat options, refer to the official documentation.
Use the created file to update the Filebeat configuration:
```
vinfra cluster filebeat config set --filename <filename> [--nodes <nodes> | --all]
```
--filename <filename>

Path to the Filebeat configuration file to upload

--nodes <nodes>

A comma-separated list of node IDs or hostnames

--all

Apply the configuration on all cluster nodes

For example, to configure Filebeat to forward the specified logs from the management node node001, run:
```
# vinfra cluster filebeat config set --nodes node001 --filename /etc/filebeat/filebeat.yml
```

To enable and start Filebeat

Enable the service:

# vinfra cluster filebeat enable [--nodes <nodes> | --all]

Start Filebeat:

# vinfra cluster filebeat start [--nodes <nodes> | --all]

To update the Filebeat configuration

Reload the Filebeat configuration:

# vinfra cluster filebeat config reload [--nodes <nodes> | --all]

Restart the service:

# vinfra cluster filebeat restart [--nodes <nodes> | --all]

To stop and disable Filebeat

Stop Filebeat:

# vinfra cluster filebeat stop [--nodes <nodes> | --all]

Disable the service:

# vinfra cluster filebeat disable [--nodes <nodes> | --all]

To delete the Filebeat configuration

Use the following command:

vinfra cluster filebeat config delete [--nodes <nodes> | --all]

--nodes <nodes>: A comma-separated list of node IDs or hostnames
--all: Delete the Filebeat configuration from all cluster nodes

For example, to delete the Filebeat configuration from the node node003, run:

# vinfra cluster filebeat config delete --nodes node003