Core storage alerts

Based on the metrics listed in Core storage metrics, the core storage alerts are generated and displayed in the admin panel.

Storage cluster alerts

Not enough cluster nodes: Cluster “<cluster_name>” has only {1,2} node(s) instead of the recommended minimum of 3. Add {2,1} or more nodes to the cluster.
Cluster is out of space: Cluster has just <free_space> TB (<free_space_in_percent>%) of physical storage space left. You may want to free some space or add more storage capacity.
Cluster is running out of physical space: There is little free physical space left on storage tier <tier> (less than 20% of free space).
Cluster is out of physical space: There is not enough free physical space on storage tier <tier> (less than 10% of free space).
Cluster has blocked or slow replication: Chunk replication is blocked or too slow.
Cluster has too many chunks: There are too many chunks in the cluster, which slows down the metadata service.
Cluster has critically high number of chunks: There are too many chunks in the cluster, which slows down the metadata service.
Cluster has too many files: There are too many files in the cluster, which slows down the metadata service.
Cluster has critically high number of files: There are too many files in the cluster, which slows down the metadata service.
Cluster has failed mount points: Some mount points stopped working and need to be recovered.
Cluster has unaligned I/O reads: I/O reads are not aligned. It may be caused by a wrongly formatted disk in a virtual machine.
Node has stuck I/O requests: Some I/O requests are stuck on <node>.
Node has failed map requests: Some map requests on <node> have failed.
CS journal is running out of space: CS journal has less than 20% of free space left.

Metadata service alerts

Not enough metadata disks: Cluster “<cluster_name>” has only one MDS. There is only one disk with the metadata role at the moment. Losing this disk will completely destroy all cluster data irrespective of the redundancy schema.
Not enough metadata disks: Cluster “<cluster_name>” requires more disks with the metadata role. Losing one more MDS will halt cluster operation.
Configuration warning: Node “<hostname>” has more than one metadata service located on it. It is recommended to have only one metadata service per node. Delete the extra metadata service(s) from this node and create them on other nodes instead.
Configuration warning: Cluster “<cluster_name>” has four metadata services. This configuration slows down the cluster performance and does not improve its availability. For a cluster of four nodes, it is enough to configure three MDSes. Delete an extra MDS from one of the cluster nodes.
Configuration warning: Cluster “<cluster_name>” has more than five metadata services. This configuration slows down the cluster performance and does not improve its availability. For a large cluster, it is enough to configure five MDSes. Delete extra MDSes from the cluster nodes.
Service failed: Metadata service #<id> is in the “<status>” state. Node: <hostname>. Disk: <disk_name>. Disk serial: <disk_serial>.
Metadata disk is out of space: Metadata disk on node “<hostname>” is running out of space (less than 10% of free space).
Metadata service has high CPU usage: Metadata service on <node> has CPU usage higher than 80%. The service may be overloaded.
Metadata service has high commit latency: Metadata service on <node> has the 95th percentile latency higher than 1 second.
Metadata service has critically high commit latency: Metadata service on <node> has the 95th percentile latency higher than 5 seconds.
Cluster has unavailable metadata services: Some metadata services are offline or have failed. Check and restart them.
Master metadata service changes too often: Master metadata service has changed more than once in 5 minutes.

Chunk service alerts

Not enough disks with storage role: Cluster “<cluster_name>” has no disks with the storage role.
Not enough disks with storage role: Cluster “<cluster_name>” has too few available CSes.
Service failed: Storage service #<id> is in the “<status>” state. Node: <hostname>. Disk: <disk_name>. Disk serial: <disk_serial>.
CS configuration is not optimal: CS#<cs_id> on tier <tier> has incorrect journalling settings.
CS configuration is not optimal: Encryption is disabled for CS#<cs_id> on tier <tier> but is enabled for other CSes on the same tier.
Storage disk is unresponsive: Disk <disk_name> (CS#<cs_id>) on node <hostname> is unresponsive. Check or replace this disk.
Disk cache settings are not optimal: Disk <disk_name> (CS#<cs_id> on node <hostname> has cache settings different from other disks of the same tier.
Cluster has offline chunk services: Some chunk services are offline. Check and restart them.
Cluster has failed chunk services: Some chunk services have failed. It may be caused by physical drive failure.
Number of CSes per device does not match configuration: Number of CSes per device on node <node> with ID <id> does not match configuration. Check your disk configuration.