S3-only
This scenario provides object storage without a compute workload. Every node runs Virtuozzo Storage and the S3 object/name services; S3 gateways run on the worker nodes. Large S3 clusters use a wide erasure-coding scheme to improve storage efficiency and support multiple storage tiers with different performance characteristics.
Cluster layout
| S3 GW | S3 GW | S3 GW | |||
| S3 OS, NS | S3 OS, NS | S3 OS, NS | S3 OS, NS | S3 OS, NS | S3 OS, NS |
| Virtuozzo Storage | Virtuozzo Storage | Virtuozzo Storage | Virtuozzo Storage | Virtuozzo Storage | Virtuozzo Storage |
| Management | Management | Management | |||
| Node1 | Node2 | Node3 | Node4 | Node5 | Node6 |
| Bond 1 (Storage) 2 × 25 GbE LACP Non-routable, MTU 9000 |
Storage · Internal management · OSTOR private (S3 internal) | ||||
| Bond 2 (Overlay + Public) 2 × 10 GbE LACP VLAN-tagged, public via NAT |
S3 public · Admin panel · Self-service · SSH | ||||
Storage policy
With 10 or more nodes, you can use a wide EC scheme such as 7+2 with the host failure domain: seven data pieces and two parity pieces across nine hosts. This configuration survives two host failures with about 29 percent storage overhead. Provision at least one more host than the scheme requires, for example 10 hosts for 7+2, so the cluster can self-heal during an outage. When EC is used, include the per-node storage-client reservation described in CPU and RAM requirements. Higher-overhead schemes such as 4+2 trade capacity efficiency for performance; 6+3 trades capacity efficiency for surviving three failures.
Three tiers
Serve different classes of object storage from separate tiers on the same cluster:
| Tier | Class | Disks | Typical EC |
|---|---|---|---|
| Premium / HPC | Lowest latency, highest IOPS | NVMe-only (no cache needed) | 4+2 (performance) |
| Mid-market | Balanced cost/performance | Fast HDD + SSD cache | 7+2 (low overhead) |
| Cold / archive | Lowest cost per TB | High-capacity SMR HDD + SSD cache | 7+2 or wider |
SMR HDDs are usable only for the storage role and only with an SSD cache. For details, see Hardware compatibility.
S3 sizing
S3 usable capacity scales with disks and network. For RAM planning, add the object, name, and gateway service reservations and, when EC is used, the storage-client reservation. The total S3 service RAM is approximately:
(CONFIGD × 256 MB) + (HOSTD × 256 MB) + (4 × S3GW × 512 MB) + (total_OS × 256 MB + total_NS × 512 MB) ÷ (S3_nodes − nodes_that_can_fail)
Worked example: 5-node S3 cluster with 3+2 erasure coding
On a management node, the base reservation is 4.5 GB RAM and 1 core for the system, 6 GB RAM and 2.4 cores for storage services, and about 1.3 GB RAM and 0.8 cores for S3 services. The EC storage client adds about 3.2 GiB RAM on a 24 GB node. This gives about 15 GB RAM and 4.2 cores before operating system headroom. Worker nodes do not require the 4.5 GB management-node reservation. After rounding up, the minimum configuration is 24 GB RAM and 8 cores. The recommended configuration is 48 GB RAM and 16 cores. S3 throughput is then limited by disks and the backend network, not by CPU or RAM.
Per-node hardware
| Resource | Management node | Worker node | ||
|---|---|---|---|---|
| Minimum | Recommended | Minimum | Recommended | |
| CPU cores (physical) | 8 | 16+ | 8 | 16+ |
| RAM (ECC, required) | 24 GB | 48 GB+ | 24 GB | 48 GB+ |
| System disk | 120 GB SSD | 2 × 480 GB SSD (RAID-1) | 120 GB SSD | 2 × 240 GB SSD (RAID-1) |
| Metadata (MDS) | On system disk (combined), first 5 nodes | On system disk (combined), first 5 nodes | ||
| Storage tiers | Per tier above (NVMe / fast-HDD+cache / SMR+cache) | Per tier above | ||
| Backend network | 2 × 10 GbE (LACP) | 2 × 25 GbE (LACP) | 2 × 10 GbE (LACP) | 2 × 25 GbE (LACP) |
| Frontend network | 10 GbE | 2 × 10 GbE (LACP) | 10 GbE | 2 × 10 GbE (LACP) |
See also