Cold Storage with S3-Compatible Stores

Many storage products speak the same language as Amazon S3. UTMStack v12 can use them as cold storage through the S3-compatible object store type. This page gives the Bucket URL format and the setup steps for MinIO, Ceph, Wasabi, Cloudflare R2 and Google Cloud Storage, plus the general rules that apply to any S3-compatible store.

For what cold storage is and how it behaves, read How UTMStack Stores Data. For Amazon S3 itself, see Cold Storage with Amazon S3.

What every store needs

Whatever the product, prepare:

1. A dedicated bucket for UTMStack. Do not share it with other applications or with another UTMStack server.

2. An access key and secret key that can list the bucket and read, write and delete objects in it. Read-only or write-only keys do not work: the event store deletes objects when records reach Delete after and when it merges blocks.

3. The Bucket URL, in this form (path style):

https://<endpoint>/<bucket>/
https://<endpoint>/<bucket>/<folder>/

The <endpoint> is the address of the storage service, with a port if it is not 443. The path must contain at least the bucket name; UTMStack refuses a URL without it ("the bucket URL must include the bucket"). UTMStack adds the final / if you leave it out.

1. Network access from the UTMStack server to the endpoint. The event store runs in a container on the server and uses the server's DNS and routes.

2. A trusted certificate if the endpoint uses https://. The certificate must be issued by a public certificate authority and match the host name in the Bucket URL. Certificates from a private or self-signed authority are refused. An http:// endpoint works, but data then travels unencrypted: use it only on a network you trust.

Then follow the UTMStack steps in Turn on cold storage in UTMStack at the end of this page.

MinIO

MinIO is object storage you run yourself.

Set it up with the MinIO client (mc). Replace the alias, host and names with yours.

# Connect mc to your MinIO server as an administrator
mc alias set myminio https://minio.example.com:9000 ADMIN_USER ADMIN_PASSWORD

# Create the bucket
mc mb myminio/utmstack-cold

Save this policy as utmstack-cold.json:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket", "s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::utmstack-cold"]
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:AbortMultipartUpload"],
      "Resource": ["arn:aws:s3:::utmstack-cold/*"]
    }
  ]
}

Then create the user and give it the policy:

mc admin policy create myminio utmstack-cold utmstack-cold.json
mc admin user add myminio utmstack-cold-user 'A-LONG-RANDOM-SECRET'
mc admin policy attach myminio utmstack-cold --user utmstack-cold-user

Use utmstack-cold-user as the Access key and its secret as the Secret key. Keep versioning and object locking off on the bucket.

Ceph (RADOS Gateway)

Ceph exposes S3 through its RADOS Gateway (RGW).

Create the user on a Ceph admin node:

radosgw-admin user create --uid=utmstack-cold --display-name="UTMStack cold storage"

The output shows the user's access_key and secret_key. Create the bucket with any S3 client using those keys, for example with the AWS CLI:

aws --endpoint-url https://rgw.example.com s3 mb s3://utmstack-cold

A bucket created by the UTMStack user belongs to it, so no extra policy is needed.

Wasabi

1. In the Wasabi console, create the bucket in the region you want. Leave versioning and object locking off.

2. Create a policy with the same content as the MinIO policy above, with your bucket name.

3. Create a sub-user with programmatic access only, attach the policy, and copy the access key and secret key.

4. Use the endpoint of the bucket's region in the Bucket URL. Wasabi lists its endpoints in its documentation; for us-east-1 the host is s3.wasabisys.com.

Cloudflare R2

1. In the Cloudflare dashboard, open R2 Object Storage and create the bucket.

2. Under Manage API tokens, create an API token with the Object Read & Write permission, limited to that bucket.

3. Copy the Access Key ID and Secret Access Key shown once on the result page, and your account ID (it is in the S3 API endpoint shown for the bucket).

4. For a bucket created with an EU jurisdiction, use the endpoint Cloudflare shows for it, https://<account-id>.eu.r2.cloudflarestorage.com.

R2 does not charge for data read out of it, which helps when analysts often search old data.

Google Cloud Storage

Google Cloud Storage accepts S3 requests through its XML API when you use HMAC keys (a key pair that works like an AWS access key).

1. 1

   Create the bucket

   In the Google Cloud console, open Cloud Storage > Buckets and click Create. Choose the location, the Standard storage class, and Uniform access control. Leave Autoclass off, keep public access prevention on, and do not set a retention policy or object versioning.

2. 2

   Create a service account

   In IAM & Admin > Service Accounts, create a service account, for example utmstack-cold-storage. It needs no project-wide role.

3. 3

   Give it access to the bucket only

   Open the bucket, then Permissions > Grant access. Add the service account with the role Storage Object User (roles/storage.objectUser). This role can list, read, create and delete objects in the bucket.

4. 4

   Create an HMAC key

   In Cloud Storage > Settings > Interoperability, under Access keys for service accounts, click Create a key for a service account and choose the service account. Copy the Access ID and the Secret; the secret is shown once.

With the gcloud CLI:

gcloud storage buckets create gs://utmstack-cold-example --location=europe-west1 
  --default-storage-class=STANDARD --uniform-bucket-level-access
gcloud iam service-accounts create utmstack-cold-storage
gcloud storage buckets add-iam-policy-binding gs://utmstack-cold-example 
  --member="serviceAccount:[email protected]" 
  --role="roles/storage.objectUser"
gcloud storage hmac create [email protected]

Other S3-compatible stores

For any other product:

• Use the path-style Bucket URL: https://<endpoint>/<bucket>/.

• Create a key that can list the bucket and read, write and delete objects, including multipart uploads.

• Check that the product supports the standard S3 calls for listing, reading, writing (including multipart upload) and deleting objects. Products that only offer a subset may fail the test when you enable cold storage, or fail later.

• If the product offers object lock, lifecycle expiry or archive tiers, leave them off for this bucket.

Turn on cold storage in UTMStack

1. 1

   Open the Cold storage card

   Go to Configure > Settings > Data Retention and scroll to Cold storage.

2. 2

   Enter the details

   Click S3-compatible object store. Enter the Bucket URL, the Access key and the Secret key. Leave Use the server's IAM role instead of keys clear: it works only with Amazon S3 on AWS.

3. 3

   Enable

   Click Enable cold storage. The event store writes a small test file to the bucket, reads it back and deletes it. If it works, the card shows In use. If not, the error starts with "the event store could not use this bucket:" and nothing is changed.

4. 4

   Choose what moves

   On the Logs card, and on Alerts if you want, set Move to cold storage after and click Save.

To check that records arrive, see Check that it works in Cold Storage with Amazon S3: the event store query and the Log Explorer search work the same way for every store. To change the keys later, enter new ones and click Update credentials.

Troubleshooting

For every UTMStack message, see Troubleshooting Retention and Cold Storage.