Backups¶

Monitoring¶

Prerequisites¶

Make sure that a bucket named 'backups' exists in OCI.

Add a credentials file to /etc/credentials with the following contexts

sudo tee /etc/credentials <<EOF
[default]
aws_access_key_id=52e5048e2668f8447e7521fbf900b862b8f4983b
aws_secret_access_key=HDWEo2ScfP2KMCh7ino5Foj+a691WVA2dja2iGwp2sk=
EOF
sudo chmod 600 /etc/credentials
sudo chown root:root /etc/credentials

Backing up Monitoring¶

The following script backs up the monitoring db and uploads it to the 'backups' Object Storage bucket.

/opt/oci-hpc/scripts/monitoring_backup.sh

Automatic Monitoring Backups¶

To set up a daily backup at midnight, add the following entry to the crontab:

0 0 * * * /opt/oci-hpc/scripts/monitoring_backup.sh

Note: Ensure the script runs successfully by checking logs or the backup bucket regularly. The script logs to /opt/oci-hpc/logs/backups/.

Clearing Monitoring Data¶

Note: Do not clear if you are backfilling. This step will delete all monitoring data.

cd /home/ubuntu/utils/victoria_metrics/yamls
sudo docker compose down
sudo rm -rf /home/ubuntu/utils/victoria_metrics/data/
sudo docker compose up -d

Restoring Monitoring data From a Backup¶

Note: Ensure Monitoring data has been cleared (see Clearing Monitoring Data section). If you are restoring a new instance of Victoria Metrics then you do not need to clear monitoring data.

# Victoria Metrics needs to be stopped before we can restore
sudo docker stop victoriametrics

# Restore
sudo /usr/bin/vmrestore \
-src=s3://backups/victoria_metrics \
-storageDataPath=/home/ubuntu/utils/victoria_metrics/data/ \
-credsFilePath=/etc/credentials \
-customS3Endpoint=axvscsfozusv.compat.objectstorage.us-sanjose-1.oraclecloud.com

# Start Victoria Metrics back up
sudo docker start victoriametrics