Updating OCI Bucket with Backup Data¶
In the beginning of every month, we need to create a new OCI bucket and move the current version of the cluster data into this bucket.
Weka filesystem can only be attached to one bucket at a time, which means at the beginning of every month, we need to repeat detach (previous month's bucket & weka) - attach (this month's bucket & weka) - synchronize (upload backup snapshots to the NEW bucket)
After upload, we ensure that only the two latest buckets are kept. The older ones are destroyed.
Object Storage Snapshot Backup¶
# *Start by creating a bucket in OCI*. If we don't already have a bucket we won't be able to successfully run the ADD OBS command below.
# NOTE: snapshots aren't removed from the bucket automatically. The bucket should be purged regularly so that old data that doesn't exist on the filesystem or in the snapshot doesn't pile up.
# At the moment we replace the bucket at the beginning of every month.
# This can be automated!
# View existing OBS
# After each step we can run these commands to view the OBS that we added and attached.
weka fs tier obs
weka fs tier s3
# Add NEW bucket's OBS
# This tells WEKA about the existence of the OCI bucket
# <bucket-namespace> - This can be found by navigating to the configuration page for the bucket we created. This can be found in the same place as the OCID.
# <bucket-name> - This is the name of the bucket we created in OCI GUI.
# The command works better if we make it inline syntax
weka fs tier s3 add <name-of-new-OBS> \
--site remote \
--hostname <bucket-namespace>.compat.objectstorage.ap-sydney-1.oraclecloud.com \
--bucket <bucket-name> \
--auth-method AWSSignature4 \
--region ap-sydney-1 \
--access-key-id 52e5048e2668f8447e7521fbf900b862b8f4983b \
--secret-key HDWEo2ScfP2KMCh7ino5Foj+a691WVA2dja2iGwp2sk= \
--protocol HTTPS
# same, in-line command
weka fs tier s3 add <name-of-new-OBS> --site remote --hostname <bucket-namespace>.compat.objectstorage.ap-sydney-1.oraclecloud.com --bucket <bucket-name> --auth-method AWSSignature4 --region ap-sydney-1 --access-key-id 52e5048e2668f8447e7521fbf900b862b8f4983b --secret-key HDWEo2ScfP2KMCh7ino5Foj+a691WVA2dja2iGwp2sk= --protocol HTTPS
# Detach OBS
# OBS looks something like OCI-month, with capitalized OCI and lowercase month
weka fs tier s3 detach default <name-of-previous-month's-OBS>
# NOTE on detaching OBS:
# Detaching won't work if there is an upload in-progress
# Check any ongoing uploads using
weka cluster task
weka fs snapshot
# Remote object status is either 1. SYNCHRONIZED 2. UPLOADING 3. NONE
# We want to look for 'UPLOADING' and either wait to finish or kill that task before proceeding with the detach
# Attach OBS
# After the OBS is added to the cluster we need to attach it to our filesystem.
weka fs tier s3 attach default <name-of-new-OBS> --mode remote
# Upload snapshot to OBS manually
# Ideally, we will set a daily / hourly synch policy in OCI.
# We can view snapshots with `weka fs snapshot`. Pick a name and upload that snapshot to the attached OBS.
weka fs snapshot upload default <snapshot-name> --site remote
# Delete old OBS
#
weka fs tier s3 delete OCI