Backup storage metrics

Metrics used for monitoring backup storage are configured in the Prometheus recording rules and can be found in the /var/lib/prometheus/rules/abgw.rules file on any node in the cluster. The most important of these metrics are described in the table:

Metric Description
FES object counters
abgw_accounts Number of accounts backup storage is currently working with (that is, number of accounts with open backup archives)
abgw_files

Number of currently open backup archives. Backup archives are open for reading and writing only during a backup operation. Other operations, such as restoring, browsing, and validation, open backup archives only for reading.

abgw_conns[proto] Number of current connections between backup storage and clients. The value is an array of counters. Details of the backup storage protocol (V1/V2) are available.
Connection counters
abgw_conns_total Total number of connections between backup storage and clients since the service startup
abgw_client_conns_cur[name] Number of currently connected clients, divided by type
abgw_client_conns_total[name] Total number of clients since the service startup, divided by type
Certificate errors and expiration times
abgw_verify_certs_errors_total[err] Total number of certificate verification errors since the service startup, divided by error type
abgw_next_certificate_expiration[path] Expiration date of backup storage certificates
abgw_cert_update_fail_total

Number of failed attempts to update the certificate revocation list. The list is required to correctly apply a new quota in Acronis Cyber Protect Cloud, when the current customer certificate is revoked and a new certificate is requested.

abgw_crl_download_fail_total Number of failed attempts to download the certificate revocation list. The list is required to correctly apply a new quota in Acronis Cyber Protect Cloud, when the current customer certificate is revoked and a new certificate is requested.
Backup storage protocol V1 request histograms and counters
abgw_read_reqs_total Number of read requests since the service startup
abgw_write_reqs_total Number of write requests since the service startup
abgw_req_errs_total[req][err] Array with request errors, divided by request type and error codes
abgw_req_latency_ms[req] Histogram with request latency
Backup storage protocol V2 request histograms and counters
abgw_v2_ireq_errs_total[req][err] Number of read requests since the service startup
abgw_v2_ireq_latency_ms[req][lat] Number of write requests since the service startup
abgw_v2_ereq_errs_total[req][err] Array with request errors, divided by request type and error codes
abgw_v2_ereq_latency_ms[req][err] Histogram with request latency
Byte counters
abgw_read_bytes_total[proxied] Number of bytes read from a disk since the service startup. The proxied parameter shows data read via a reverse proxy.
abgw_write_bytes_total[proxied] Number of bytes written to a disk since the service startup. The proxied parameter shows data written via a reverse proxy.
abgw_write_rollback_bytes_total

Size of data overwritten by backup storage per client's request when backup storage could not confirm to the client that data was already written. The metric is used only for the backup storage protocol V1 and legacy backup clients.

File operation and I/O operation metrics
abgw_file_lookup_errs_total[err] Number of failed attempts to open files or find already open files, divided by error codes
abgw_fop_latency_ms_bucket[fop][proxied][err] Histogram with the sum of file operation latency, divided by operation type (read, write, sync, stat), proxied or not, by error number, and other file operations
abgw_iop_latency_ms_bucket[iop][proxied][err] Histogram with I/O operation latency, divided by operation type, proxied or not, and by error number
abgw_io_limiting_failures_total[type] Number of failed I/O requests to backup storage since the service startup, due to poor performance of the underlying storage
abgw_iop_wd_timeouts[iop] Number of file operations that take more than two minutes, divided by operation type
Migration metrics
abgw_account_pull_errs_total[err]

Number of failed attempts to retrieve the account list by the destination backup storage from the source backup storage before the migration start

abgw_nr_files_to_pull Number of files to migrate from the source backup storage to the destination backup storage (includes all files for which migration is not completed)
abgw_pull_backlog_bytes Number of bytes on the source backup storage that are not yet migrated to the destination backup storage
abgw_pull_progress_bytes_total Number of bytes on the destination backup storage that are already migrated from the source backup storage since the service startup
abgw_file_migration_source_open_errs_total[err] Number of failed attempts to open files for migration on the source backup storage since the service startup
abgw_file_migration_source_read_errs_total[err] Number of failed attempts to read files for migration on the source backup storage since the service startup
abgw_nr_accounts_pull_pending Number of accounts that are awaiting migration
abgw_nr_accounts_pull_started Number of accounts that are in the process of migration
abgw_nr_accounts_pull_errors Number of accounts that have migration errors in at least one file
Object storage and geo-replication metrics
abgw_push_backlog_bytes[ostor, replica] Number of bytes to be written to the object destination storage, or to the secondary cluster in case of geo-replication
abgw_push_progress_bytes_total[ostor, replica] Number of bytes written to the object destination storage, or to the secondary cluster in case of geo-replication. This metric helps to understand the speed of data replication or copying.
abgw_push_replica_errs_total[err] Number of failed attempts to write files to the object destination storage, or to the secondary cluster in case of geo-replication, since the service startup, divided by error type
abgw_replica_integrity_checks_fail_total

Number of corrupted replicas on the secondary cluster since the service startup

abgw_file_replica_auto_errs_total[err] Number of geo-replication errors for new files (created after configuring geo-replication) since the service startup, divided by error type
abgw_file_replica_open_errs_total[err] Number of failed attempts by the primary cluster to open files for writing on the secondary cluster since the service startup, divided by error code
abgw_rm_file_push_errs_total[err] Number of errors occurred when removing files from the secondary cluster since the service startup, divided by error type. A secondary replica is deleted after disabling replication on the primary cluster.
abgw_push_replica_total_size_by_brand[brand] Total disk space occupied by replication-enabled files on the primary cluster, grouped by brand ID.
abgw_push_progress_by_brand[brand] Current number of bytes successfully replicated from the primary cluster to the secondary one, grouped by brand ID.
Object destination storage metrics
abgw_ostor_used_space_bytes Space size used by all backup archives, including data and unused space, on the object destination storage
abgw_nr_ostor_sequence_mismatch_total Number of files failed to be opened by backup storage due to their version mismatch on the object destination storage
abgw_ostor_garbage_bytes Unused space size inside all backup archives that is not yet physically cleaned up on the object destination storage
Container archive validation results
abgw_containers_validate_segments_fail_total

Number of archives with failed validation (segments) on the NFS and object destination storage

abgw_containers_validate_trees_fail_total Number of archives with failed validation (trees) on the NFS and object destination storage
Other metrics
abgw_append_throttle_delay_ms_total Total sum of delays injected since the service startup. The metric helps to understand if throttling is enabled for backup storage.
abgw_iop_ebusy Number of I/O errors for open file operations since the service startup

Histogram metrics with the "_bucket" suffix have corresponding metrics ending with "_sum" and "_counter", for example:

  • abgw_iop_latency_ms_bucket shows the current measurement for I/O operation latency per bucket
  • abgw_iop_latency_ms_count shows the total sum of all measurements for I/O operation latency per bucket
  • abgw_iop_latency_ms_sum shows the number of stored measurements for I/O operation latency per bucket