Creating the S3 cluster

Limitations

  • After the S3 cluster deployment, you can change only the replication redundancy scheme for data. Changing the encoding redundancy scheme for data is disabled, because it may decrease cluster performance. Re-encoding demands a significant amount of cluster resources for a long period of time. If you still want to change the redundancy scheme for data, contact the technical support team.

Prerequisites

  • A clear understanding of the concept Storage policies.
  • The storage cluster has at least one disk with the Storage role.
  • Ensure that each node to join the object storage cluster has the TCP port 443 (HTTPS) or TCP port 80 (HTTP) open for outgoing and incoming Internet connections.

To set up object storage services on cluster nodes

Admin panel

  1. On the Infrastructure > Networks screen, make sure that the OSTOR private and S3 public traffic types are added to the networks you intend to use.
  2. Open the Storage services > S3 screen, and then click Create S3 storage.
  3. On the Nodes step, select nodes to add to the S3 storage, and then click Next. To create highly available S3 storage, select at least three nodes.
  4. On the Storage policies step, define storage policies for S3 data and metadata:

    • In the Data storage policy section, select the desired tier, failure domain, and data redundancy mode for storing S3 data. To benefit from high availability, select a mode other than No redundancy and a failure domain other than Disk.

    • In the Metadata storage policy section, select the desired tier for storing S3 metadata, which includes NS and OS journals. It is highly recommended to place metadata on a faster storage tier than is used for data, to improve the service performance.

  5. On the DNS step, specify an external DNS name for the S3 storage. For example, s3storage.example.com. End users will use this DNS name and the TCP port 443 (HTTPS) or TCP port 80 (HTTP) to access the S3 data. Then, click Next.

    DNS load balancing can be used for test purposes only. For production, use an external load balancer.

  6. On the Protocol step, select an S3 endpoint protocol: HTTP, HTTPS, or both.

    It is recommended to use only HTTPS for production deployments.

    If you selected the HTTPS protocol, do one of the following:

    • Select Upload a certificate, specify the prepared SSL certificate, and then specify the SSL key or passphrase (for PKCS#12 files).

      You need to acquire a key and a trusted wildcard SSL certificate for endpoint’s bottom-level domain. For example, the endpoint s3storage.example.com would need a wildcard certificate for *.s3storage.example.com with the subject alternative name s3storage.example.com.

    • Select Generate a certificate, to get a self-signed certificate for HTTPS evaluation purposes.

      • S3 geo-replication requires a certificate from a trusted authority. It does not work with self-signed certificates.
      • To access the data in the S3 cluster via a browser, add the self-signed certificate to browser’s exceptions.

    Then, click Next.

  7. On the Summary step, review the configuration, and then click Create.

To check if the S3 storage is successfully deployed and can be accessed by users, visit https://<S3_DNS_name> or http://<S3_DNS_name> in your browser. You should receive the following XML response:

<Error>
<Code>AccessDenied</Code>
<Message/>
</Error>

To start using the S3 storage, you will also need to create at least one S3 user.

Command-line interface

Use the following command:

vinfra service s3 cluster create [--tier {0,1,2,3}] [--failure-domain {0,1,2,3,4}]
                                 [--replicas <norm> | --encoding <M>+<N>] [--metadata-tier {0,1,2,3}]
                                 [--self-signed | --no-ssl | --cert-file <cert_file>]
                                 [--insecure] [--key-file <key_file>] [--password]
                                 --nodes <nodes> --s3gw-domain <domain> --s3gw-count <s3gw_count>
                                 --os-count <os_count> --ns-count <ns_count>
--tier {0,1,2,3}
Storage tier (default: 0)
--failure-domain {0,1,2,3,4}
Storage failure domain (default: 0)
--replicas <norm>

Storage replication mapping in the format:

  • norm: the number of replicas to maintain (default: 1)
--encoding <M>+<N>

Storage erasure encoding mapping in the format:

  • M: the number of data blocks
  • N: the number of parity blocks
--metadata-tier {0,1,2,3}
Storage tier
--self-signed
Generate a new self-signed certificate (default)
--no-ssl
Do not generate a self-signed certificate
--cert-file <cert_file>
Path to a file with the new certificate
--insecure
Allow insecure connections in addition to secure ones (only used with the --cert-file and --self-signed options)
--key-file <key_file>
Path to a file with the private key (only used with the --cert-file option)
--password
Read certificate password from stdin (only used with the --cert-file option)
--nodes <nodes>
A comma-separated list of node hostnames or IDs
--s3gw-domain <domain>
DNS name S3 endpoint
--s3gw-count <s3gw_count>
Number of S3 gateways
--os-count <os_count>
Amount of OS services in S3 cluster
--ns-count <ns_count>
Amount of NS services in S3 cluster

Increasing the number of gateways, NS, and OS services also increases the resource requirements. To learn more about CPU and RAM reservations for the S3 services, refer to General requirements.

For example, to create the S3 cluster from nodes node001 and node002 with a self-signed certificate, run:

# vinfra service s3 cluster create --nodes node001,node002 --tier 0 --failure-domain 1 --encoding 1+2 \
--metadata-tier 1 --self-signed --s3gw-domain dns.example.com

This command also specifies the tier, failure domain, redundancy mode, and domain name.

You can view the S3 storage details in the vinfra service s3 show output:

# vinfra service s3 show
+-----------------+--------------------------------------------+
| Field           | Value                                      |
+-----------------+--------------------------------------------+
| failure_domain  | 1                                          |
| id              | 0100000000000002                           |
| metadata_policy | failure_domain: 1                          |
|                 | redundancy:                                |
|                 |   m: 1                                     |
|                 |   n: 2                                     |
|                 |   type: raid6                              |
|                 | tier: 1                                    |
| name            | cluster1                                   |
| nodes           | - id: ca334b1d-20a1-1241-96a5-eb9acadb8ecd |
|                 | - id: ab36b523-91dc-e78d-53a7-88baed44541e |
| np              |                                            |
| nusers          | 0                                          |
| protocol        | scheme: https                              |
| redundancy      | m: 1                                       |
|                 | n: 2                                       |
|                 | type: raid6                                |
| s3gw_domain     | dns.example.com                            |
| tier            | 0                                          |
+-----------------+--------------------------------------------+