Configuring S3 load balancing

To efficiently distribute traffic between multiple S3 nodes with gateways, you can use an external load balancer such as HAProxy. HAProxy is an open-source, high-performance solution that provides high availability, load balancing, and proxying for TCP- and HTTP-based applications. It is scalable, supports multiple load-balancing algorithms, session persistence, and SSL certificate management.

Load balancing for S3

HAProxy supports several load-balancing strategies that can be used for S3 backends:

Round robin

Requests are distributed evenly across all backend servers. This is simple and effective for similar-performance servers.

balance roundrobin

Least connections

HAProxy forwards new connections to the server with the fewest active connections. Ideal for servers with uneven workloads.

balance leastconn

Source (IP hash)

Requests from the same client IP are sent to the same server, which can improve caching efficiency.

balance source

We recommend using least connections because S3 operations can vary in duration, and it ensures that busy nodes do not get overloaded.

Health checks

Health checks are critical to ensure HAProxy routes traffic only to healthy nodes. If a node becomes unresponsive, HAProxy automatically redirects traffic to healthy nodes. The health-check settings include the following:

HTTP checks

Periodically send an HTTP request to a specific endpoint (for example, /?ostor-health) to verify the server is responsive.

option httpchk
http-check send meth GET uri /?ostor-health

Check interval

Controls how often HAProxy checks each backend server.

server s3.source 10.136.21.80:80 id 101 check inter 5s

Retries and timeout

Configure the number of failed attempts before marking a server as down and the timeouts for connecting or waiting for a response.

retries 3
timeout connect 30s
timeout server 30s

Logging health checks

Enables monitoring of health-check results for easier troubleshooting.

option log-health-checks

Example HAProxy configuration

global
        maxconn                 10000
        stats socket /tmp/haproxy.socket level admin expose-fd listeners
        uid                     80
        gid                     80
        nbthread                1
        hard-stop-after         15m
        chroot                  /tmp/haproxy_chroot
        daemon
        tune.ssl.default-dh-param 2048
        server-state-file       /tmp/haproxy_server_state

frontend s3_frontend
        bind                    10.136.20.226:443 name 10.136.20.226:443 ssl crt-list /var/etc/haproxy/s3_frontend.crt_list
        mode                    tcp
        log                     global
        timeout client          30s
        default_backend         s3_backend_ipvANY

backend s3_backend_ipvANY
        mode                    tcp
        id                      100
        log                     global
        option                  log-health-checks
        option                  httpchk
        http-check              send meth GET uri /?ostor-health
        balance                 leastconn
        timeout connect         30s
        timeout server          30s
        retries                 3
        load-server-state-from-file global
        server                  s3.source 10.136.21.80:80 id 101 check inter 5s

For a full list of available configuration options and advanced health-check settings, refer to the HAProxy documentation.