How to use Borgmatic to backup PostgreSQL in Kubernetes
There are many goodies in the internets, but not much good documentation so that usage of them would be frictionless. Here is a short writeup on backuping up Mastodon instance database, running on Kubernetes with Borgmatic, but it could be used for any generic database and/or file path, that is supported by Borg backup and Borgmatic.
For the destination/target repository I am using Borg Base, but it should work with any Borg compatible ssh repo.
There are some issues that I might solve in the future, namely – storing sensitive information in safer way, but for the moment I just wanted to make backups work. If you will do it – please let me know and I will update my setup.
First things first, we need: – Kubernetes – Mastodon deployment in its own namespace – Setup SSH keys – Setup Healtchecks – Setup Borgbase – Install Borgmatic via Helm Chart, using our values
You should be runnning something already
You should have deployed Mastodon already in its own namespace. Instructions should work for any generic deployment that uses PostgreSQL or any other supported database.
Setup SSH keys
You need to generate ssh key and save both parts – private and public. Private will go to our Borgmatic setup and public – to Borgbase.
ssh-keygen -t ed25519 -f /tmp/id_ed25519
This can be done on any computer that has ssh-keygen utility.
/tmp/id_ed25519will contain the private part and
/tmp/id_ed25519.pub – the public. You don't need the files themselves, just contents.
This is optional, but do it if you do not use Borgbase – get https://healthchecks.io/ account – it is free for small numbers of checks. Borgbase also has the same functionality, so you might skip it if you chose to use Borgbase.
Again – you can use whatever repository server that supports Borg, but I found Borgbase having all the features I require for the cheap price. Add public ssh key part to keys and make sure that repository is tied to this key. Save the repository address.
Install Borgmatic via Helm Chart, using our values
I have used this Chart – https://charts.gabe565.com/charts/borgmatic/ – it is great stuff, just had some issues figuring out how it works. I am kinda slow learner – I make a lot of assumptions that result in being me wrong most of the time. The trick was to find good values for values.yaml:
# # IMPORTANT NOTE # # This chart inherits from our common library chart. You can check the default values/options here: # https://github.com/bjw-s/helm-charts/blob/main/charts/library/common/values.yaml # image: # -- image repository repository: ghcr.io/borgmatic-collective/borgmatic # -- image pull policy pullPolicy: IfNotPresent # -- image tag tag: 1.7.14 controller: # -- Set the controller type. Valid options are `deployment` or `cronjob`. type: deployment cronjob: # -- Only used when `controller.type: cronjob`. Sets the backup CronJob time. schedule: 0 * * * * # -- Only used when `controller.type: cronjob`. Sets the CronJob backoffLimit. backoffLimit: 0 # -- environment variables. [[ref]](https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables) # @default -- See [values.yaml](./values.yaml) env: # -- Borg host ID used in archive names # @default -- Deployment namespace BORG_HOST_ID: "" BORG_PASSPHRASE: ---GENERATE THIS WITH pwgen OR SIMILAR TOOL--- PGPASSWORD: ---PostgresSQL db password--- # MYSQL_PWD: # MONGODB_PASSWORD: persistence: # -- Configure persistence settings for the chart under this key. # @default -- See [values.yaml](./values.yaml) data: enabled: false retain: true # storageClass: "" # accessMode: ReadWriteOnce # size: 1Gi subPath: - path: borg-repository mountPath: /mnt/borg-repository - path: config mountPath: /root/.config/borg - path: cache mountPath: /root/.cache/borg # -- Configure SSH credentials for the chart under this key. # @default -- See [values.yaml](./values.yaml) ssh: name: borgmatic-ssh enabled: true type: configMap mountPath: /root/.ssh/ readOnly: false defaultMode: 0600 configMaps: # -- Configure Borgmatic container under this key. # @default -- See [values.yaml](./values.yaml) ssh : enabled: true data: id_ed25519: | -----BEGIN OPENSSH PRIVATE KEY----- --- PASTE YOUR PRIVATE KEY HERE--- -----END OPENSSH PRIVATE KEY----- known_hosts: | --- paste the output of ssh-keyscan borg-repository-address --- config: enabled: true data: # -- Crontab crontab.txt: |- 0 1 * * * PATH=$PATH:/usr/bin /usr/local/bin/borgmatic --stats -v 0 2>&1 # -- Borgmatic config. [[ref]](https://torsion.org/borgmatic/docs/reference/configuration) # @default -- See [values.yaml](./values.yaml) config.yaml: | location: # List of source directories to backup. source_directories: - /etc/ <- any directory you want as we are concerned only with db backup # Paths of local or remote repositories to backup to. repositories: - ---BORG REPOSITORY URL--- retention: # Retention policy for how many backups to keep. keep_daily: 7 keep_weekly: 4 keep_monthly: 6 consistency: # List of checks to run to validate your backups. checks: - name: repository - name: archives frequency: 2 weeks hooks: # Databases to dump and include in backups. postgresql_databases: - name: mastodon_production hostname: hostname-of-mastodon-postgresql-db username: mastodon # Third-party services to notify you if backups aren't happening. healthchecks: --- healtcheck url ---
helm install borgmatic gabe565/borgmatic -f values.yaml -n mastodon kubectl rollout status deployment.apps/borgmatic -n mastodon kubectl exec -i deployment.apps/borgmatic -n mastodon -- borgmatic init --encryption repokey-blake2 kubectl exec -it deployment.apps/borgmatic -n mastodon -- borgmatic create --stats
All of these commands should be completed without errors