4.3. Accessing Virtuozzo Storage Clusters via S3-like Object Storage¶

4.3.1. About Object Storage¶

Object storage is a storage architecture that enables managing data as objects (like in key-value storage) as opposed to files in file systems or blocks in block storage. Except data, each object has name (i.e. full path to object) that describes it and also a unique identifier that allows finding said object in the storage. Object storage is optimized for storing billions of objects, in particular for application back-end storage, static web content hosting, online storage services, big data, and backups. All of these uses are enabled by object storage thanks to a combination of very high scalability and data availability and consistency.

Compared to other types of storage, the key difference of S3 object storage is that parts of an object cannot be modified, so if the object changes a new version of it is spawned instead. This approach is extremely important for maintaining data availability and consistency. First of all, changing an object as a whole eliminates the issue of conflicts. That is, the object with the latest timestamp is considered to be the current version and that is it. As a result, objects are always consistent, i.e. their state is relevant and appropriate.

Another feature of object storage is eventual consistency. Eventual consistency does not guarantee that reads are to return the new state after the write has been completed. Readers can observe the old state for an undefined period of time until the write is propagated to all the replicas (copies). This is very important for storage availability as geographically distant data centers may not be able to perform data update synchronously (e.g., due to network issues) and the update itself may also be slow as awaiting acknowledges from all the data replicas over long distances can take hundreds of milliseconds. So eventual consistency helps hide communication latencies on writes at the cost of the probable old state observed by readers. However, many use cases can easily tolerate it.

4.3.1.1. Object Storage Infrastructure¶

The infrastructure of Virtuozzo Object Storage consists of the following entities: object servers, name servers, S3 gateways, and the block level backend.

• Object server (OS) stores actual object data (contents) received from S3 gateway. It stores its own data in block storage with built-in high availability.

• Name server stores object metadata received from S3 gateway. Metadata includes object name, size, ACL (access control list), location, owner, and such. Name server (NS) also stores its own data in block storage with built-in high availability.

• S3 gateway (GW) is a data proxy between object storage services and end users. It receives and handles Amazon S3 protocol requests and uses nginx Web server for external connections. S3 gateway handles S3 user authentication and ACL checks. It has no data of its own (i.e. is stateless).

• Block level backend is block storage with high availability of services and data. Since all object storage services run on hosts, no virtual environments (or respective licenses) are required for object storage.

4.3.2. Deploying Object Storage¶

This chapter describes deploying object storage on top of a ready Virtuozzo Storage cluster. As a result you will create a setup like shown on the figure. Note that not all cluster nodes have to run object storage services. The choice should be based on workload and hardware configurations.

To set up object storage services, do the following:

1. Plan the S3 network. Like a Virtuozzo Storage cluster, an object storage cluster needs two networks:

   • An internal network in which NS, OS, and GW will interact. These services will generate traffic similar in amount to the total (incoming and outgoing) S3 user traffic. If this is not going to be much, it is reasonable to use the same internal network for both object storage and Virtuozzo Storage. If, however, you expect that object storage traffic will compete with Virtuozzo Storage traffic, it is reasonable to have S3 traffic go through the user data network (i.e. datacenter network). Once you choose a network for S3 traffic, you determine which IP addresses can be used while adding cluster nodes.

   • An external (public) network through which end users will access the S3 storage. Standard HTTP and HTTPS ports must be open in this network.

   An object storage cluster is almost completely independent on base block storage (like all access points, including virtual environments and iSCSI). Object and name servers keep their data in the Virtuozzo Storage cluster in the same way as virtual environments, iSCSI, and other services do. So the OS and NS services depend on vstorage-mount (client) and can only work when the cluster is mounted. Unlike them, gateway is a stateless service that has no data. It is thus independent on vstorage-mount and can theoretically be run even on nodes where the Virtuozzo Storage cluster is not mounted. However, for simplicity, we recommend creating gateways on nodes with object and name servers.

   Object and name servers also utilize the standard high availability means of Virtuozzo Storage (i.e. the shaman service). Like virtual environments and iSCSI, OS and NS are subscribed to HA cluster events. However, unlike other services, S3 cluster components cannot be managed (tracked and relocated between nodes) by shaman. Instead, this is done by the S3 configuration service that is subscribed to HA cluster events and notified by shaman whether nodes are healthy and can run services. For this reason, S3 cluster components are not shown in shaman top output.

   Gateway services which are stateless are never relocated and their high availability is not managed by the Virtuozzo Storage cluster. Instead, a new gateway service is created when necessary.

2. Make sure that each node that will run OS and NS services is in the high availability cluster. You can add nodes to HA cluster with the shaman join command.

3. Install the vstorage-ostor package on each cluster node.

   # yum install vstorage-ostor
   

4. Create a cluster configuration on one of the cluster nodes where object storage services will run. It is recommended to create 10 NS and 10 OS services per each node. For example, if you are going to use five nodes, you will need 50 NS and 50 OS. Run this command on the first cluster node.

   # ostor-ctl create -r /var/lib/ostor/configuration -n <IP_addr>
   

   Where <IP_addr> is the node’s IP address (that belongs to the internal S3 network) that the configuration service will listen on.

   You will be asked to enter and confirm a password for the new object storage (it can be the same as your Virtuozzo Storage cluster password). You will need this password to add new nodes.

   The configuration service will store the cluster configuration locally in /var/lib/ostor/configuration. In addition, <IP_addr> will be stored in /<storage_mount>/<ostor_dir>/control/name (<ostor_dir> is the directory in the cluster with object storage service files). If the first configuration service fails (and the ostor-ctl get-config command stops working), replace the IP address in /<storage_mount>/<ostor_dir>/control/name with that of a node running a healthy configuration service (created on step 6).

5. Launch the configuration service.

   # systemctl start ostor-cfgd.service
   # systemctl enable ostor-cfgd.service
   

6. Add at least two more configuration services for redundancy (to have at least three in total). A configuration service is only required for adding and removing nodes to and from the S3 cluster and does not affect operation of S3 services and their high availability. So a failure of a configuration service is not critical for the S3 cluster. However, it is still undesirable and we recommend creating several configuration services so at least one is always up.

   To add one more configuration service, run the following commands on a node where object storage services will run. Repeat to create the required number of configuration services.

   # ostor-ctl join -n <remote_IP_addr> -a <local_IP_addr>
   # systemctl start ostor-cfgd.service
   # systemctl enable ostor-cfgd.service
   

   Where <remote_IP_addr> is <IP_addr> from step 4.

   Each added configuration service will store the cluster configuration locally in /var/lib/ostor/configuration.

7. Initialize the new object storage on the first node. The <ostor_dir> directory will be created in the root of your cluster.

   # ostor-ctl init-storage -n <IP_addr> -s <cluster_mount_point>
   

   You will need to provide the IP address and object storage password specified on step 3.

8. Add to the DNS public IP addresses of nodes that will run GW services. You can configure the DNS to enable access to your object storage via a hostname, and to have the S3 endpoint receive virtual hosted-style REST API requests with URIs like http://bucketname.s3.example.com/objectname.

   After configuring DNS, make sure that DNS resolver for your S3 access point works from client machines.

   Note

   Only buckets with DNS-compatible names can be accessed with virtual hosted-style requests. For more details, see Bucket and Key Naming Policies.

   Below is an example of a DNS zones configuration file for the BIND DNS server:

   ;$Id$
   $TTL 1h  @  IN   SOA    ns.example.com. s3.example.com. (
                           2013052112      ; serial
                           1h      ; refresh
                           30m     ; retry
                           7d      ; expiration
                           1h )    ; minimum
                   NS      ns.example.com.
   $ORIGIN s3.example.com
   h1 IN   A 10.29.1.95
           A 10.29.0.142
           A 10.29.0.137
   * IN    CNAME    @
   

   This configuration instructs the DNS to redirect all requests with URI http://s3.example.com and its subdomains (http://*.s3.example.com/*) to one of the endpoints listed in resource record h1 (10.29.1.95, 10.29.0.142 or 10.29.0.137) in a cyclic (round-robin) manner.

9. Add nodes where object storage services will run to the configuration.

   Note

   Adding nodes to existing clusters is done in the similar way by performing steps 8-12.

   To do this, run the ostor-ctl add-host command on every such node:

   # ostor-ctl add-host -r /var/lib/ostor/configuration --hostname <name> --roles OBJ
   

   You will need to provide the object storage password set on step 3.

   Note

   If you want the object storage agent service to listen on an internal IP address, add the option -H <internal_IP_address> to the command above.

10. Create a new S3 volume with the desired number of NS and OS:

    # ostor-ctl add-vol --type OBJ -s <cluster_mount_point> --os-count <OS_num> \
    --ns-count <NS_num> --vstorage-attr "failure-domain=host,tier=0,replicas=3"
    

    Where:

    • <NS_num> and <OS_num> are the numbers of NS and OS

    • failure-domain=host, tier=0, replicas=3 parameters set volume’s failure domain, tier, and redundancy mode (for more details, see Cluster Parameters Overview).

    The command will return the ID for the created volume. You will need it on the next step.

11. Create S3 gateway instances on chosen nodes with Internet access and external IP addresses. It is recommended to create 4 GWs per node.

    Note

    For security reasons, make sure that only nginx can access the external network and that S3 gateways only listen on internal IP addresses.

    # ostor-ctl add-s3gw -a <internal_IP_address>:<port> -V <volume_ID>
    

    Where:

    • <internal_IP_address> is the internal IP address of the node with the gateway

    • <port> (mandatory) is an unused port unique for each GW instance on the node

    • <volume_ID> is the ID of the volume you created on the previous step (it can also be obtained from ostor-ctl get-config)

    For example:

    # ostor-ctl add-s3gw -a 127.0.0.1:9001 -V 0100000000000001
    # ostor-ctl add-s3gw -a 127.0.0.1:9002 -V 0100000000000001
    # ostor-ctl add-s3gw -a 127.0.0.1:9003 -V 0100000000000001
    # ostor-ctl add-s3gw -a 127.0.0.1:9004 -V 0100000000000001
    

12. Launch object storage agent on each cluster node added to the object storage configuration.

    # systemctl start ostor-agentd.service
    # systemctl enable ostor-agentd.service
    

13. Make sure NS and OS services are bound to the nodes.

    By default agents will try to assign NS and OS services to the nodes automatically in a round-robin manner. However, manual assignment is required if a new host has been added to the configuration, or if the current configuration is not optimized (for details, see Manually Binding Services to Nodes.

    You can check the current binding configuration with the ostor-ctl agent-status command. For example:

    # ostor-ctl agent-status
    TYPE     SVC_ID               STATUS          UPTIME  HOST_ID            ADDRS
    S3GW     8000000000000009     ACTIVE             527  fcbf5602197245da   127.0.0.1:9090
    S3GW     8000000000000008     ACTIVE             536  4f0038db65274507   127.0.0.1:9090
    S3GW     8000000000000007     ACTIVE             572  958e982fcc794e58   127.0.0.1:9090
    OS       1000000000000005     ACTIVE             452  4f0038db65274507   10.30.29.124:39746
    OS       1000000000000004     ACTIVE             647  fcbf5602197245da   10.30.27.69:56363
    OS       1000000000000003     ACTIVE             452  4f0038db65274507   10.30.29.124:52831
    NS       0800000000000002     ACTIVE             647  fcbf5602197245da   10.30.27.69:56463
    NS       0800000000000001     ACTIVE             452  4f0038db65274507   10.30.29.124:53044
    NS       0800000000000000     ACTIVE             647  fcbf5602197245da   10.30.27.69:37876
    

14. On each node with GW instances, install nginx that will serve S3 requests from end users:

    # yum install nginx
    

    Use the ostor-configure-nginx tool to configure nginx for S3. There are two configuration options:

    • create that creates a new configuration

    • update that updates the existing upstream configuration; use it after changing GW configuration.

    For the initial nginx configuration, use create. For example, for HTTP:

    # ostor-configure-nginx create -D s3.mydomain.com -p 80
    

    Where s3.mydomain.com is the S3 endpoint domain and 80 is the port for nginx to listen to.

    To configure HTTP with an SSL certificate for the S3 endpoint domain and its subdomains, specify the certificate and key. For example:

    # ostor-configure-nginx create -D s3.mydomain.com -p 443 **ssl **ssl-cert file.cert **ssl-key file.key
    

    The configuration file will be created at /etc/nginx/conf.d/ostor-s3.conf. It will handle FastCGI redirection to local GW instances.

15. Launch nginx:

    # systemctl start nginx.service
    # systemctl enable nginx.service
    

16. Add nodes that are in the HA cluster but run no S3 services to the S3 cluster. That is, make sure that all nodes in the HA cluster are also in the S3 cluster. This is required for high availability to work correctly.

    # ostor-ctl add-host -n <IP_addr>
    # systemctl start ostor-agentd.service
    # systemctl enable ostor-agentd.service
    

The object storage is deployed. Now you can add S3 users with the ostor-s3-admin tool as described in Creating S3 Users.

To check that installation has been successful or just monitor object storage status, use the ostor-ctl get-config command. For example:

# ostor-ctl get-config
07-08-15 11:58:45.470 Use configuration service 'ostor'
SVC_ID             TYPE  URI
8000000000000006   S3GW  svc://1039c0dc90d64607/?address=127.0.0.1:9000
0800000000000000     NS  vstorage://cluster1/ostor/services/0800000000000000
1000000000000001     OS  vstorage://cluster1/ostor/services/1000000000000001
1000000000000002     OS  vstorage://cluster1/ostor/services/1000000000000002
1000000000000003     OS  vstorage://cluster1/ostor/services/1000000000000003
1000000000000004     OS  vstorage://cluster1/ostor/services/1000000000000004
8000000000000009   S3GW  svc://7a1789d20d9f4490/?address=127.0.0.1:9000
800000000000000c   S3GW  svc://7a1789d20d9f4490/?address=127.0.0.1:9090

# ostor-ctl bind -H 4f0038db65274507 -S 0800000000000001 -S 1000000000000003 -S 1000000000000005

# ostor-ctl get-config
SVC_ID             TYPE  URI
0800000000000000     NS  vstorage://stor1/s3-data/services/0800000000000000
0800000000000001     NS  vstorage://stor1/s3-data/services/0800000000000001
0800000000000002     NS  vstorage://stor2/s3-data/services/0800000000000002
1000000000000003     OS  vstorage://stor1/s3-data/services/1000000000000003
1000000000000004     OS  vstorage://stor2/s3-data/services/1000000000000004
1000000000000005     OS  vstorage://stor1/s3-data/services/1000000000000005
HOST_ID            HOSTNAME      URI
0fcbf5602197245da  host510:2530  vstorage://stor1/s3-data
4f0038db65274507   host511:2530  vstorage://stor1/s3-data
958e982fcc794e58   host512:2530  vstorage://stor2/s3-data
953e976abc773451   host513:2530  vstorage://stor2/s3-data

4.3.3. Managing S3 Users¶

The concept of S3 user is one of the base concepts of object storage along with those of object and bucket (container for storing objects). Amazon S3 protocol uses permissions model based on access control lists (ACLs) where each bucket and each object is assigned an ACL that lists all users with access to the given resource and the type of this access (read, write, read ACL, write ACL). The list of users includes entity owner assigned to every object and bucket at creation. Entity owner has extra rights compared to other users, for example, bucket owner is the only one who can delete that bucket.

User model and access policies implemented in Virtuozzo Object Storage comply with the Amazon S3 user model and access policies.

User management scenarios in Virtuozzo Object Storage are largely based on the Amazon Web Services user management and include the following operations: create, query, delete users as well as generate, revoke user access key pairs.

You can manage users with the ostor-s3-admin tool. To do this, you will need to know the ID of the volume that the users are in. You can obtain it with the ostor-ctl get-config command. For example:

# ostor-ctl get-config -n 10.94.97.195
VOL_ID             TYPE     STATE
0100000000000002   OBJ     READY
...

# ostor-s3-admin create-user -e user@email.com -V 0100000000000002
UserEmail:user@email.com
UserId:a49e12a226bd760f
KeyPair[0]:S3AccessKeyId:a49e12a226bd760fGHQ7
KeyPair[0]:S3SecretAccessKey:HSDu2DA00JNGjnRcAhLKfhrvlymzOVdLPsCK2dcq
Flags:none

# ostor-s3-admin query-users -V 0100000000000002
      S3 USER ID      S3 ACCESS KEY ID              S3 SECRET ACCESS KEY  S3 USER EMAIL
bf0b3b15eb7c9019  bf0b3b15eb7c9019I36Y                               ***  user2@abc.com
d866d9d114cc3d20  d866d9d114cc3d20G456                               ***  user1@abc.com
                  d866d9d114cc3d20D8EW                               ***
e86d1c19e616455                      -                                 -  user3@abc.com

# ostor-s3-admin query-users -V 0100000000000002 -a -X
<?xml version="1.0" encoding="UTF-8"?><QueryUsersResult><Users><User><Id>a49e12a226bd760f</Id><Email>user@email.com</Email><Keys><OwnerId>0000000000000000</OwnerId><KeyPair><S3AccessKeyId>a49e12a226bd760fGHQ7</S3AccessKeyId><S3SecretAccessKey>HSDu2DA00JNGjnRcAhLKfhrvlymzOVdLPsCK2dcq</S3SecretAccessKey></KeyPair></Keys></User><User><Id>d7c53fc1f931661f</Id><Email>user@email.com</Email><Keys><OwnerId>0000000000000000</OwnerId><KeyPair><S3AccessKeyId>d7c53fc1f931661fZLIV</S3AccessKeyId><S3SecretAccessKey>JL7gt1OH873zR0Fzv8Oh9ZuA6JtCVnkgV7lET6ET</S3SecretAccessKey></KeyPair></Keys></User></Users></QueryUsersResult>

# ostor-s3-admin query-user-info -e user@email.com -V 0100000000000002
Query user: user id=d866d9d114cc3d20, user email=user@email.com
Key pair[0]: access key id=d866d9d114cc3d20G456,
secret access key=5EAne6PLL1jxprouRqq8hmfONMfgrJcOwbowCoTt
Key pair[1]: access key id=d866d9d114cc3d20D8EW,
secret access key=83tTsNAuuRyoBBqhxMFqHAC60dhKHtTCCkQe54zu

# ostor-s3-admin disable-user -e user@email.com -V 0100000000000002

# ostor-s3-admin delete-user -i bf0b3b15eb7c9019 -V 0100000000000002
Deleted user: user id=bf0b3b15eb7c9019

# ostor-s3-admin gen-access-key -e user@email.com -V 0100000000000002
Generate access key: user id=d866d9d114cc3d20, access key id=d866d9d114cc3d20D8EW,
secret access key=83tTsNAuuRyoBBqhxMFqHAC60dhKHtTCCkQe54zu

# ostor-s3-admin revoke-access-key -e user@email.com -k de86d1c19e616455YIPU -V 0100000000000002
Revoke access key: user id=de86d1c19e616455, access key id=de86d1c19e616455YIPU

4.3.4. Managing Object Storage Buckets¶

All objects in Amazon S3-like storage are stored in containers named buckets. Buckets are addressed by names that are unique in the given object storage, so an S3 user of that object storage cannot create a bucket that has the same name as a different bucket in the same object storage. Buckets are used to:

• Group and isolate objects from those in other buckets.

• Provide ACL management mechanisms for objects in them.

• Set per-bucket access policies, for example, versioning in the bucket.

You can manage buckets with the ostor-s3-admin tool as well as S3 API third-party S3 browsers like CyberDuck or DragonDisk. The ostor-s3-admin tool is to be used by object storage administrators, so these commands do not include any authentication or authorization checks. It is recommended to use standard Amazon S3 API commands first.

To manage buckets via CLI, you will need to know the ID of the volume that the buckets are in. You can obtain it with the ostor-ctl get-config command. For example:

# ostor-ctl get-config -n 10.94.97.195
VOL_ID             TYPE     STATE
0100000000000002   OBJ     READY
<...>

4.3.4.1. Managing Buckets with CyberDuck¶

4.3.4.1.1. Creating Buckets¶

To create a new S3 bucket with CyberDuck, do the following:

1. Click Open Connection.

2. Specify the following parameters:

   • The external DNS name for the S3 endpoint that you specified when creating the S3 cluster.

   • The Access Key ID and the Secret Access Key of an object storage user (see Creating S3 Users).

   

   By default, the connection is established over HTTPS. To use CyberDuck over HTTP, you must install a special S3 profile.

3. Once the connection is established, click File > New Folder.

   

4. Specify a name for the new bucket, and then click Create.

   Note

   It is recommended to use bucket names that comply with DNS naming conventions. For more information on bucket naming, see Bucket and Key Naming Policies.

The new bucket will appear in CyberDuck and you can manage it and upload files into it.

4.3.4.1.2. Managing Bucket Versions¶

Versioning is a way of keeping multiple variants of an object in the same bucket. You can use versioning to preserve, retrieve, and restore every version of every object stored in your S3 bucket. With versioning, you can easily recover from both unintended user actions and application failures. For more information about bucket versioning, refer to the Amazon documentation.

Bucket versioning is turned by default. You can turn it on from a third-party S3 browser by selecting a checkbox in the bucket property. For example:

4.3.4.1.3. Listing Bucket Contents¶

You can list bucket contents with a web browser. To do this, visit the URL that consists of the external DNS name for the S3 endpoint that you specified when creating the S3 cluster and the bucket name. For example, mys3storage.example.com/mybucket.

Note

You can also copy the link to bucket contents by right-clicking it in CyberDuck, and then selecting Copy URL.

# ostor-s3-admin list-all-buckets -V 0100000000000002
Total 3 buckets
BUCKET                OWNER             CREATION_DATE  VERSIONING     TOTAL SIZE, BYTES
bucket1    968d1a79968d1a79  2015-08-18T09:32:35.000Z        none                  1024
bucket2    968d1a79968d1a79  2015-08-18T09:18:20.000Z     enabled                     0
bucket3    968d1a79968d1a79  2015-08-18T09:22:15.000Z   suspended               1024000

# ostor-s3-admin list-all-buckets -X
<?xml version="1.0" encoding="UTF-8"?><ListBucketsResult><Buckets><Bucket><Name>bucker2</Name><Owner>d7c53fc1f931661f</Owner><CreationDate>2017-04-03T17:11:44.000Z</CreationDate><Versioning>none</Versioning><Notary>off</Notary><TotalSize>0</TotalSize></Bucket><Bucket><Name>bucket1</Name><Owner>d7c53fc1f931661f</Owner><CreationDate>2017-04-03T17:11:33.000Z</CreationDate><Versioning>none</Versioning><Notary>off</Notary><TotalSize>0</TotalSize></Bucket></Buckets></ListBucketsResult>

# ostor-s3-admin list-all-buckets -i d7c53fc1f931661f
BUCKET   OWNER             CREATION_DATE             VERSIONING  TOTAL_SIZE NOTARY NOTARY_PROVIDER
bucker2  d7c53fc1f931661f  2017-04-03T17:11:44.000Z  none        0          off    0

# ostor-s3-admin query-bucket-info -b bucket1 -V 0100000000000002
BUCKET   OWNER             CREATION_DATE             VERSIONING  TOTAL_SIZE
bucket1  d339edcf885eeafc  2017-12-21T12:42:46.000Z  none        0

ACL: d339edcf885eeafc: FULL_CONTROL

# ostor-s3-admin change-bucket-owner -b bucket1 -i bf0b3b15eb7c9019 -V 0100000000000002
Changed owner of the bucket bucket1. New owner bf0b3b15eb7c9019

# ostor-s3-admin delete-bucket -b bucket1 -V 0100000000000002

4.3.5. Best Practices for Using Object Storage¶

This chapter describes recommendations on using various features of Virtuozzo Object Storage. These recommendations are called to help you enable additional functionality or improve convenience or performance of Virtuozzo Object Storage.

4.3.6. Appendices¶

This section provides reference information related to Virtuozzo Object Storage.