Performing node maintenance

Whenever you need to perform service operations on a cluster node, place it in the maintenance mode. When you do so, the node stops allocating new chunks of storage data, but continues to handle I/O operations for the core storage services such as MDS, CS, and cache. They will not, however, be used to allocate new data, so placing the node in maintenance may reduce the free space in the storage cluster. All storage services on the node will continue serving data unless the node goes offline. Other node’s services (compute, Backup Gateway, iSCSI, S3, and NFS) can either be relocated or left as is during maintenance.

If a service cannot be evacuated from the node for some reason, entering maintenance will be halted. You will need to decide on how to proceed: exit or force maintenance.

Once the node is in the maintenance mode, you can shut it down and perform service operations on it. Once you are done, power on the node and return it to operation in the admin panel.

If needed, you can place more nodes in the maintenance mode, one at a time. The cluster can tolerate multiple nodes in maintenance simultaneously as long as the other cluster nodes have enough resources to accommodate evacuated workloads. This also applies during a cluster update, which can be performed when the cluster already has nodes in maintenance. If nodes that have entered the maintenance mode are online, they are updated together with the other cluster nodes, but do not exit maintenance after the update is complete. If a node goes offline while in maintenance, the cluster becomes degraded, thus blocking an update. You can wait until this node is up again or release it from the cluster.

Limitations

The last management node cannot be placed in the maintenance mode.
You cannot evacuate a service from the last operational node where this service is deployed. For example, VMs that are hosted on the last compute node cannot be relocated from this node and will be ignored during maintenance.
Suspended VMs cannot be relocated from the node and will be ignored.
Nodes in maintenance can only be returned to operation or released.

Prerequisites

A clear understanding of cluster self-healing, which is explained in Сluster rebuilding.
You have five MDS services in the storage cluster. In this case, when a node running the MDS service is shut down during maintenance, the cluster can survive the failure of another node.
If the node hosts virtual machines, the other compute nodes have enough resources to accommodate these VMs.
If the node hosts iSCSI targets, the iSCSI initiators are configured to use multiple IP addresses from the same target group.
If the node runs an S3 gateway, its IP addresses are removed from the DNS records of the S3 access points. Otherwise, some of the S3 clients may experience connection timeouts.

To place a node in the maintenance mode

Admin panel

On the Infrastructure > Nodes screen, click the line with the desired node.
On the node right pane, click Enter maintenance.
In the Enter maintenance window, select to Evacuate or Ignore the following workloads during maintenance:
- Block storage. iSCSI target groups are highly available, with multiple targets running on different nodes. When the node enters maintenance, the target it hosts is stopped and the preferred path is moved to another node in the target group within 60 seconds. Thus, the service is not interrupted during maintenance.
- Compute. Evacuating virtual machines from the node means migrating them live one by one to other compute nodes. If you select to ignore them, they will continue running until you reboot or shut down the node. In this case, they will be stopped, resulting in downtime. They also will not be started automatically once the node is up again.
- S3. You can evacuate S3 services from this node to other nodes in the S3 cluster or ignore them. In the latter case, they will continue running until you reboot or shut down the node, resulting in downtime. They will be started automatically once the node is up again.
- NFS. You can evacuate NFS services from this node to other nodes in the NFS cluster or ignore them. In the latter case, they will continue running until you reboot or shut down the node, resulting in downtime. They will be started automatically once the node is up again.
- ABGW. This service is highly available, with multiple instances spread across different nodes. Placing this node in the maintenance mode will stop one of the instances, but the others will continue working, so the service will not be interrupted.
Start data healing. Cluster self-healing is an automatic restoration of storage cluster data that becomes unavailable when a storage node (or a disk) goes offline. If this happens during maintenance, self-healing is delayed (for 30 minutes by default) to save cluster resources. If the node goes back online before the delay ends, self-healing is not necessary.

You can manually configure the replication timeout by setting the mds.wd.offline_tout_mnt parameter, in milliseconds, with the vstorage -c <cluster_name> set-config command.
If the node has non-redundant chunks of data, you will see the Relocate non-redundant data option. Select it to move the non-redundant data to other storage nodes. Otherwise, this data will become unavailable if the node goes offline. The data may also be temporarily moved to another tier if the current one is full.
Click Enter.

Command-line interface

Use the following commands:

Start maintenance precheck for a node. For example:
```
# vinfra node maintenance precheck node001
```

View the maintenance status. For example:

# vinfra node maintenance status node001
+-----------+------------------------------------------+
| Field     | Value                                    |
+-----------+------------------------------------------+
| node_id   | c3b2321a-7c12-8456-42ce-8005ff937e12     |
| params    | alua_mode: suspend                       |
|           | compute_mode: evacuate                   |
|           | iscsi_mode: evacuate                     |
|           | nfs_mode: evacuate                       |
|           | s3_mode: evacuate                        |
|           | storage_mode: suspend                    |
| precheck  | flow: completed                          |
|           | id: c15bf919-9a81-45b1-8fef-5f626e68f957 |
|           | result:                                  |
|           | - has_resources: true                    |
|           |   relocation_is_possible: true           |
|           |   resources: null                        |
|           |   service: node                          |
|           |   service_is_available: true             |
|           | - has_resources: false                   |
|           |   relocation_is_possible: true           |
|           |   resources: null                        |
|           |   service: iscsi                         |
|           |   service_is_available: true             |
|           | - has_resources: true                    |
|           |   relocation_is_possible: true           |
|           |   resources:                             |
|           |     failed: []                           |
|           |   service: alua                          |
|           |   service_is_available: true             |
|           | - has_resources: true                    |
|           |   relocation_is_possible: true           |
|           |   resources: null                        |
|           |   service: compute                       |
|           |   service_is_available: true             |
|           | - has_resources: false                   |
|           |   relocation_is_possible: null           |
|           |   resources: null                        |
|           |   service: nfs                           |
|           |   service_is_available: false            |
|           | - has_resources: true                    |
|           |   relocation_is_possible: false          |
|           |   resources: null                        |
|           |   service: s3                            |
|           |   service_is_available: true             |
|           | state: success                           |
|           | updated_at: '2021-11-01T11:09:41.331926' |
| resources |                                          |
| state     | completed                                |
| task      |                                          |
+-----------+------------------------------------------+

The output above shows that the node has relocatable iSCSI and compute services, as well as the S3 service that cannot be relocated.

Start the node maintenance by running the command:
```
vinfra node maintenance start [--iscsi-mode <mode>] [--compute-mode <mode>]
                              [--s3-mode <mode>] [--storage-mode <mode>]
                              [--alua-mode <mode>] [--nfs-mode <mode>] <node>
```
--iscsi-mode <mode>

Ignore ISCSI evacuation during maintenance (ignore).

--compute-mode <mode>

Ignore compute evacuation during maintenance (ignore).

--s3-mode <mode>

Ignore S3 evacuation during maintenance (ignore).

--storage-mode <mode>

Ignore storage evacuation during maintenance (ignore).

--alua-mode <mode>

Ignore Block Storage target groups during maintenance (ignore).

--nfs-mode <mode>

Ignore NFS evacuation during maintenance (ignore).

<node>

Node ID or hostname

For example, to start maintenance for the node node001 without evacuating its S3 service, run:
```
# vinfra node maintenance start node001 --s3-mode ignore
```

To proceed after entering maintenance halted

On the Infrastructure > Nodes screen, click the line with the desired node.
On the node right pane, click Enter maintenance.
Select the desired action:
- Select Exit maintenance to return all services on the node to their normal state.
- Select Force maintenance to stop the services that could not be evacuated during node reboot or shutdown.
Click Continue.

To return a node to operation

Admin panel

On the Infrastructure > Nodes screen, click the line with the desired node.
On the node right pane, click Exit maintenance.
In the confirmation window, click Yes.

Command-line interface

Use the following command:

vinfra node maintenance stop <node> [--ignore-compute]

<node>: Node ID or hostname
--ignore-compute: Ignore compute resources while returning a node to operation

For example, to stop maintenance for the node node001, run:

# vinfra node maintenance stop node001