Enabling S3 cross-region replication

Cross-region replication (CRR) enables automatic, asynchronous copying of objects across S3 buckets in different regions. Buckets that are configured for CRR can be owned by the same user. Objects may be replicated to a single destination bucket or multiple destination buckets.

To enable CRR, you need to add replication configuration to your source bucket. The minimum configuration must provide the destination buckets where you want to replicate objects and a user with the role that enables replicating objects on your behalf.

Limitations

  • CRR only supports copying new S3 objects after it is enabled.

Prerequisites

  • S3 clusters are created, as described in Creating the S3 cluster.
  • To be able to use the AWS command-line tool, install and configure it first, as described in the official documentation Install/Update and Setup.

To set up CRR replication

  1. Create two or more S3 buckets, one bucket in the source S3 cluster and one or multiple buckets in the destination S3 cluster. You can also use buckets that already exist in your S3 clusters. For example, to create the source and destination buckets, use:

    # ostor-s3-cmd create-bucket -e http://s3.ostor_source -k f6995711d730d7dc3T0T -s 3tNgp4aGJYjtDG8UFdzHoFR8HKMyV5qD0DWiGMBP -b source
    # ostor-s3-cmd create-bucket -e http://s3.ostor_destination -k c460186c7b91a79f -s z4Fw8p8y2jTdjusTme2uBaLDe6p69F5yUPFuHHz8 -b destination

    Where:

    • -e is the URL of the S3 cluster where to create a bucket.
    • -k and -s are the access and secret keys of the S3 user.
    • -b specifies a bucket name.
  2. Enable versioning for these buckets. For example:

    # ostor-s3-cmd enable-versioning -e http://s3.ostor_source -k f6995711d730d7dc3T0T -s 3tNgp4aGJYjtDG8UFdzHoFR8HKMyV5qD0DWiGMBP -b source
    # ostor-s3-cmd enable-versioning -e http://s3.ostor_destination -k c460186c7b91a79f -s z4Fw8p8y2jTdjusTme2uBaLDe6p69F5yUPFuHHz8 -b destination
  3. Create a replication configuration file. For example, the replication.conf file may look as follows:

    {
      "Role": "arn:aws:iam::f6995711d730d7dc:role/service-role/s3crr_role",
      "Rules": [
        {
          "ID": "rep-rule",
          "Status": "Enabled",
          "Priority": 1,
          "Filter": {},
          "Destination": {
            "Bucket": "arn:aws:s3:::destination"
          },
          "DeleteMarkerReplication": {
            "Status": "Disabled"
          }
        }
      ]
    }

    Where:

    • Role is the ID of the user that is used to replicate objects on your behalf. In the example above, it is f6995711d730d7dc.
    • Rule specifies which objects to replicate and where to store the replicas.
    • ID is a unique identifier for the rule. The value must be up to 255 characters long.
    • Status specifies whether the rule is enabled. Valid values are Enabled or Disabled.
    • Priority indicates which rule has precedence whenever two or more replication rules conflict.
    • Bucket is the name of the bucket where you want to store the results.
    • Destination is a container for information about the replication destination and its configuration.
    • DeleteMarkerReplication specifies whether to replicate delete markers. If the Filter element is specified, you must also include the DeleteMarkerReplication element.
  4. Configure the source bucket for CRR by specifying the replication configuration file. For example:

    # ostor-s3-cmd put-bucket-replication-config -e http://s3.ostor_source -k f6995711d730d7dc3T0T -s 3tNgp4aGJYjtDG8UFdzHoFR8HKMyV5qD0DWiGMBP \
    -b source -c replication.conf -D http://s3.ostor_destination -K c460186c7b91a79f -S z4Fw8p8y2jTdjusTme2uBaLDe6p69F5yUPFuHHz8

    Where:

    • -e is the URL of the source S3 cluster.
    • -k and -s are the access and secret keys of the user that is used to replicate objects on your behalf. In the example above, it is f6995711d730d7dc.
    • -b is the name of the source bucket.
    • -c specifies the replication configuration file.
    • -D is the URL of the destination S3 cluster.
    • -K and -S are the access and secret keys of the user that has access to the destination buckets.