Managing high availability configuration

Once the management node high availability (HA) is created, it includes three nodes and protects your infrastructure from a failure of one management node. Additionally, you can expand the HA configuration up to five nodes, to keep your cluster operational even if two management nodes fail simultaneously.

Though it is possible to have four nodes in the management node HA, such a configuration should be considered intermediary, as it is highly recommended to have an odd number of nodes included.

The HA node that is used by the backend services is called Primary. After creating the HA configuration, the current management node usually becomes the primary one. You can change the primary node by promoting another node in the HA configuration to primary. When the primary node changes, the backend services move from the current management node to the selected HA node. During this operation, which may take up to five minutes, the admin and self-service panels, as well as the compute API, are unavailable.

Depending on the number of nodes, management of the HA configuration may differ:

With a three-node configuration, that is, with the minimum number of nodes, removing a node from the HA configuration is not possible without destroying the management node HA. If one of the HA nodes fails, you need to replace it with a healthy one.
With a five-node configuration, you can remove up to two nodes without destroying the management node HA. If one of the HA nodes fails, you can either replace it with a healthy one in one iteration, or you can remove the failed node, and then add another one instead.

When replacing a node in the HA configuration, you need to consider its role in the compute cluster:

A node that is added to the compute cluster and used to host virtual machines has the Worker role. In the vinfra command-line tool, this role is called compute.
A management node or a node that is included into the HA configuration has the Controller role. Such nodes are automatically added to the compute cluster, but they are not used to host VMs.
A node that is not added to the compute cluster is called a storage node.

The replacement option depends on the Worker role, that is, whether a node belongs to the compute cluster. Worker nodes can only be replaced with other nodes from the compute cluster. Non-worker nodes have two options, they can be replaced with both compute and storage nodes. The figure below shows all possible replacement options in the five-node HA configuration.

Destroying the HA configuration is required for network migration that enables changing network configuration.

Limitations

The primary node cannot be replaced in the HA configuration.
If one or more management nodes enter the maintenance mode, a failure of another management node may affect high availability of the cluster.
While the management node HA is being destroyed, management of the compute cluster may be unavailable.

Prerequisites

A clear understanding of the limitations listed in High availability and the compute cluster.
The HA configuration is created, as described in Enabling management node high availability.

To add nodes to the HA configuration

Admin panel

Go to Settings > System settings > Management node high availability.
In the High availability nodes section, click Options, and then click Add node.
In the Add node window, select one or two nodes to be added to the HA configuration, and then click Add.

Command-line interface

Use the following command:

vinfra cluster ha node add --nodes <nodes>

--nodes <nodes>

A comma-separated list of node IDs or hostnames

For example, to add the nodes node001 and node002 to the HA configuration, run:

# vinfra cluster ha node add --nodes node001,node002

To change the primary node in the HA configuration

Admin panel

Go to Settings > System settings > Management node high availability.
Click the ellipsis icon next to the node that you wish to make primary in the HA configuration, and then click Promote to primary node.
In the confirmation window, click Promote.

During this operation, you will be logged out of the system due to migration of the backend services to the selected node.

Command-line interface

Use the following command:

vinfra cluster ha switch-primary <node>

<node>: Name or ID of the new primary node

For example, to promote the node node001 to primary, run:

# vinfra cluster ha switch-primary node001

To replace nodes in the HA configuration

Admin panel

Go to Settings > System settings > Management node high availability.
Click the ellipsis icon next to the node that you wish to replace in the HA configuration, and then click Replace.
In the Replace node window, select the node that will be added into the HA configuration instead of the removed node, and then click Replace.

Once an offline node is removed from the high availability configuration, it will remain so after becoming available again.

Command-line interface

Use the following command:

vinfra cluster ha update [--virtual-ip <network:ip>] [--nodes <nodes>] [--force]

--virtual-ip <network:ip>

HA configuration mapping in the format:

network: network to include in the HA configuration (must include at least one of these traffic types: Internal management, Admin panel, Self-service panel, or Compute API).
ip: virtual IP address that will be used in the HA configuration.

Specify this option multiple times to create a HA configuration for multiple networks.

--nodes <nodes>

A comma-separated list of node IDs or hostnames

--force

Skip checks for minimal hardware requirements

For example, to update the management node HA configuration, that is, include the nodes node001, node002, and node005, run:

# vinfra cluster ha update --nodes node001,node002,node005

To remove nodes from the HA configuration

Admin panel

Go to Settings > System settings > Management node high availability.
Click the ellipsis icon next to the node that you wish to remove from the HA configuration, and then click Remove.
In the confirmation window, click Remove.

Command-line interface

Use the following command:

vinfra cluster ha node remove [--force] <node>

--force: Skip the compute cluster state and forcibly remove the node(s). This option is required when removing multiple nodes or offline nodes.; Node ID(s) or hostname(s) to be removed. Note that the HA configuration must have at least 3 nodes to be operational.

For example, to remove the nodes node002 and node005 from the HA configuration, run:

# vinfra cluster ha node remove node002 node005 --force

To destroy the HA configuration

Admin panel

Go to Settings > System settings > Management node high availability.
In the High availability nodes section, click Options, and then click Destroy HA configuration.
In the confirmation window, click Destroy.

Once the high availability configuration is destroyed, you can log in to the admin panel at the IP address of the management node and on the same port 8888.

Command-line interface

Use the following command:

vinfra cluster ha delete