Backup and Restore

Backing Up Kubernetes

Wayfinder has been tested against a backup and restore strategy using Velero. For more information, see the Velero documetation. Velero lets you create scheduled backups of all Kubernetes resources inside the cluster it is installed into, and archives the backup data into cloud storage, such as an S3 or GCP Cloud Storage bucket. This procedure does not back up the Wayfinder database.

note

The following procedures demostrate manual steps for illustrative purposes. In production, this should be automated via a CI pipeline.

Create an S3 bucket to hold the backups

Example:

export AWS_REGION=<aws region>
export BUCKET_NAME=<name of s3 bucket>

# Create a bucket to hold the backups
$ aws s3api create-bucket \
  --bucket ${BUCKET_NAME} \
  --acl private \

# Ensure the bucket contents is encrypted with KMS or AES256
$ aws s3api put-bucket-encryption \
  --bucket ${BUCKET_NAME} \
  --server-side-encryption-configuration '{
  "Rules": [
    {
      "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "AES256"
      }
    }
  ]
}'

Create a policy that can read/write to the S3 bucket

Example:

cat <<EOF > policy.json
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action": "s3:ListAllMyBuckets",
         "Resource":"arn:aws:s3:::*"
      },
      {
         "Effect":"Allow",
         "Action":["s3:ListBucket","s3:GetBucketLocation"],
         "Resource":"arn:aws:s3:::${BUCKET_NAME}"
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:PutObject",
            "s3:PutObjectAcl",
            "s3:GetObject",
            "s3:GetObjectAcl",
            "s3:DeleteObject"
         ],
         "Resource":"arn:aws:s3:::${BUCKET_NAME}/*"
      }
   ]
}
EOF

# create the iam policy
$ aws iam create-policy --policy-name wf-backup --policy-document file://policy.json

Create an IAM user and attach the policy

Example:

# create a iam group to apply the policy
$ aws iam create-group --name wf-backups

# attach the wayfinder backup policy to the group
$ aws iam attach-group-policy --name wf-backups --policy-arn <arn-from-before>

# create the user used for backups
$ aws iam create-user --user-name wf-backups
$ aws iam create-access-key --user-name wf-backups

# attach the user to the group
$ aws iam add-user-to-group wf-backups --user-name wf-backups

Create the credentials.aws containing the AWS access keys

Example:

[default]
aws_access_key_id = <access_key>
aws_secret_access_key = <access_secret>

Install Velero into the Wayfinder management cluster

The example below installs Velero into the velero namespace, creating the various apigroups, backup location and deployment.

$ velero install --provider aws \
  --plugins velero/velero-plugin-for-aws:v1.1.0 \
  --bucket ${BUCKET_NAME} \
  --secret-file ./credentials.aws \
  --backup-location-config region=${AWS_REGION} \
  --snapshot-location-config region=${AWS_REGION}

Create a scheduled backup for Wayfinder cluster

Example:

$ velero schedule create wf-backup \
  --schedule="0 */1 * * *" \ # The frequency of the backups
  --ttl=120h               \ # How long before the backup can be garbage collected

At this point Velero is installed and performing backups on the Wayfinder management cluster on the hour.

Restore from backups

The following assumes you are restoring the backup to a newly built cluster, though the process is the same regardless. You must install Velero into that cluster. Once you have the Velero service up and running, check the backups using aws s3 ls s3://${BUCKET_NAME}.

To restore from backup:

1. Ensure the backup location has been marked as ReadOnly for security:

$ kubectl patch backupstoragelocation default \
  --namespace velero \
  --type merge \
  --patch '{"spec":{"accessMode":"ReadOnly"}}'

2. Once you have the backup name, run the following to restore:

velero create restore --from-backup NAME

You can watch the restore status by running:

velero get restore

1. Ensure the Wayfinder namespace is restored in the cluster and check the status of the pods:

kubectl -n wf get po