Performing node maintenance

Whenever you need to perform service operations on a cluster node, place it in the maintenance mode. When you do so, the node stops allocating new chunks of storage data, but continues to handle I/O operations for the core storage services such as MDS, CS, and cache. They will not, however, be used to allocate new data, so placing the node in maintenance may reduce the free space in the storage cluster. All CSes on the node will continue serving data unless the node goes offline. Other node’s services (compute, Backup Gateway, iSCSI, S3, and NFS) can either be relocated or left as is during maintenance.

If a service cannot be evacuated from the node for some reason, entering maintenance will be halted. You will need to decide on how to proceed: exit or force maintenance.

Once the node is in the maintenance mode, you can shut it down and perform service operations on it. Once you are done, power on the node and return it to operation in the admin panel.

Limitations

  • Suspended VMs cannot be relocated from the node and will be ignored.
  • Nodes in maintenance can be returned to operation or released.

Prerequisites

  • A clear understanding of cluster self-healing, which is explained in Сluster rebuilding.
  • You have five MDS services in the storage cluster. In this case, when a node running the MDS service is shut down during maintenance, the cluster can survive the failure of another node.
  • If the node hosts virtual machines, the other compute nodes have enough resources to accommodate these VMs.
  • If the node hosts iSCSI targets, the iSCSI initiators are configured to use multiple IP addresses from the same target group.
  • If the node runs an S3 gateway, its IP addresses are removed from the DNS records of the S3 access points. Otherwise, some of the S3 clients may experience connection timeouts.

To place a node in the maintenance mode

Admin panel

  1. On the Infrastructure > Nodes screen, click the line with the desired node.
  2. On the node right pane, click Enter maintenance.
  3. In the Enter maintenance window, select to Evacuate or Ignore the following workloads during maintenance:

    • Block storage. iSCSI target groups are highly available, with multiple targets running on different nodes. When the node enters maintenance, the target it hosts is stopped and the preferred path is moved to another node in the target group within 60 seconds. Thus, the service is not interrupted during maintenance.
    • Compute. Evacuating virtual machines from the node means migrating them live one by one to other compute nodes. If you select to ignore them, they will continue running until you reboot or shut down the node. In this case, they will be stopped, resulting in downtime. They also will not be started automatically once the node is up again.
    • S3. You can evacuate S3 services from this node to other nodes in the S3 cluster or ignore them. In the latter case, they will continue running until you reboot or shut down the node, resulting in downtime. They will be started automatically once the node is up again.
    • NFS. You can evacuate NFS services from this node to other nodes in the NFS cluster or ignore them. In the latter case, they will continue running until you reboot or shut down the node, resulting in downtime. They will be started automatically once the node is up again.
    • ABGW. This service is highly available, with multiple instances spread across different nodes. Placing this node in maintenance mode will stop one of the instances, but the others will continue working, so the service will not be interrupted.

  4. Start data healing. Cluster self-healing is an automatic restoration of storage cluster data that becomes unavailable when a storage node (or a disk) goes offline. If this happens during maintenance, self-healing is delayed (for 30 minutes by default) to save cluster resources. If the node goes back online before the delay ends, self-healing is not necessary.

    You can manually configure the replication timeout by setting the mds.wd.offline_tout_mnt parameter, in milliseconds, with the vstorage -c <cluster_name> set-config command.

  5. If the node has non-redundant chunks of data, you will see the Relocate non-redundant data option. Select it to move the non-redundant data to other storage nodes. Otherwise, this data will become unavailable if the node goes offline. The data may also be temporarily moved to another tier if the current one is full.
  6. Click Enter.

Command-line interface

Use the following commands:

  1. Start maintenance precheck for a node. For example:

    # vinfra node maintenance precheck node001
    
  2. View the maintenance status. For example:

    # vinfra node maintenance status node001
    +-----------+------------------------------------------+
    | Field     | Value                                    |
    +-----------+------------------------------------------+
    | node_id   | c3b2321a-7c12-8456-42ce-8005ff937e12     |
    | params    |                                          |
    | precheck  | flow: completed                          |
    |           | id: c15bf919-9a81-45b1-8fef-5f626e68f957 |
    |           | result:                                  |
    |           | - has_resources: true                    |
    |           |   relocation_is_possible: true           |
    |           |   resources: null                        |
    |           |   service: node                          |
    |           |   service_is_available: true             |
    |           | - has_resources: false                   |
    |           |   relocation_is_possible: true           |
    |           |   resources: null                        |
    |           |   service: iscsi                         |
    |           |   service_is_available: true             |
    |           | - has_resources: false                   |
    |           |   relocation_is_possible: true           |
    |           |   resources:                             |
    |           |     failed: []                           |
    |           |   service: alua                          |
    |           |   service_is_available: false            |
    |           | - has_resources: true                    |
    |           |   relocation_is_possible: false          |
    |           |   resources: null                        |
    |           |   service: compute                       |
    |           |   service_is_available: true             |
    |           | - has_resources: false                   |
    |           |   relocation_is_possible: null           |
    |           |   resources: null                        |
    |           |   service: nfs                           |
    |           |   service_is_available: false            |
    |           | - has_resources: true                    |
    |           |   relocation_is_possible: false          |
    |           |   resources: null                        |
    |           |   service: s3                            |
    |           |   service_is_available: true             |
    |           | state: success                           |
    |           | updated_at: '2021-11-01T11:09:41.331926' |
    | resources |                                          |
    | state     | idle                                     |
    | task      |                                          |
    +-----------+------------------------------------------+

    The output above shows that the node has the compute and S3 services that cannot be relocated.

  3. Start the node maintenance by running the command:

    vinfra node maintenance start [--iscsi-mode <mode>] [--compute-mode <mode>]
                                  [--s3-mode <mode>] [--storage-mode <mode>]
                                  [--alua-mode <mode>] [--nfs-mode <mode>] <node>
    
    --iscsi-mode <mode>
    Ignore ISCSI evacuation during maintenance (ignore).
    --compute-mode <mode>
    Ignore compute evacuation during maintenance (ignore).
    --s3-mode <mode>
    Ignore S3 evacuation during maintenance (ignore).
    --storage-mode <mode>
    Ignore storage evacuation during maintenance (ignore).
    --alua-mode <mode>
    Ignore Block Storage target groups during maintenance (ignore).
    --nfs-mode <mode>
    Ignore NFS evacuation during maintenance (ignore).
    <node>
    Node ID or hostname

    For example, to start maintenance for the node node001 without evacuating its S3 and compute services, run:

    # vinfra node maintenance start node001 --s3-mode ignore --compute-mode ignore
    

To proceed after entering maintenance halted

  1. On the Infrastructure > Nodes screen, click the line with the desired node.
  2. On the node right pane, click Enter maintenance.
  3. Select the desired action:
    • Select Exit maintenance to return all services on the node to their normal state.
    • Select Force maintenance to stop the services that could not be evacuated during node reboot or shutdown.
  4. Click Continue.

To return a node to operation

Admin panel

  1. On the Infrastructure > Nodes screen, click the line with the desired node.
  2. On the node right pane, click Exit maintenance.

Command-line interface

Use the following command:

vinfra node maintenance stop <node> [--ignore-compute]
<node>
Node ID or hostname
--ignore-compute

Ignore compute resources while returning a node to operation

For example, to stop maintenance for the node node001, run:

# vinfra node maintenance stop node001