How to use Borgmatic to backup PostgreSQL in Kubernetes
There are many goodies in the internets, but not much good documentation so that usage of them would be frictionless. Here is a short writeup on backuping up Mastodon instance database, running on Kubernetes with Borgmatic, but it could be used for any generic database and/or file path, that is supported by Borg backup and Borgmatic.
For the destination/target repository I am using Borg Base, but it should work with any Borg compatible ssh repo.
There are some issues that I might solve in the future, namely – storing sensitive information in safer way, but for the moment I just wanted to make backups work. If you will do it – please let me know and I will update my setup.
First things first, we need: – Kubernetes – Mastodon deployment in its own namespace – Setup SSH keys – Setup Healtchecks – Setup Borgbase – Install Borgmatic via Helm Chart, using our values
Kubernetes
You should be runnning something already
Mastodon deployment
You should have deployed Mastodon already in its own namespace. Instructions should work for any generic deployment that uses PostgreSQL or any other supported database.
Setup SSH keys
You need to generate ssh key and save both parts – private and public. Private will go to our Borgmatic setup and public – to Borgbase.
ssh-keygen -t ed25519 -f /tmp/id_ed25519
This can be done on any computer that has ssh-keygen utility. /tmp/id_ed25519will contain the private part and /tmp/id_ed25519.pub – the public. You don't need the files themselves, just contents.
Setup Healthchecks
This is optional, but do it if you do not use Borgbase – get https://healthchecks.io/ account – it is free for small numbers of checks. Borgbase also has the same functionality, so you might skip it if you chose to use Borgbase.
Setup Borgbase
Again – you can use whatever repository server that supports Borg, but I found Borgbase having all the features I require for the cheap price. Add public ssh key part to keys and make sure that repository is tied to this key. Save the repository address.
Install Borgmatic via Helm Chart, using our values
I have used this Chart – https://charts.gabe565.com/charts/borgmatic/ – it is great stuff, just had some issues figuring out how it works. I am kinda slow learner – I make a lot of assumptions that result in being me wrong most of the time. The trick was to find good values for values.yaml:
#
# IMPORTANT NOTE
#
# This chart inherits from our common library chart. You can check the default values/options here:
# https://github.com/bjw-s/helm-charts/blob/main/charts/library/common/values.yaml
#
image:
# -- image repository
repository: ghcr.io/borgmatic-collective/borgmatic
# -- image pull policy
pullPolicy: IfNotPresent
# -- image tag
tag: 1.7.14
controller:
# -- Set the controller type. Valid options are `deployment` or `cronjob`.
type: deployment
cronjob:
# -- Only used when `controller.type: cronjob`. Sets the backup CronJob time.
schedule: 0 * * * *
# -- Only used when `controller.type: cronjob`. Sets the CronJob backoffLimit.
backoffLimit: 0
# -- environment variables. [[ref]](https://borgbackup.readthedocs.io/en/stable/usage/general.html#environment-variables)
# @default -- See [values.yaml](./values.yaml)
env:
# -- Borg host ID used in archive names
# @default -- Deployment namespace
BORG_HOST_ID: ""
BORG_PASSPHRASE: ---GENERATE THIS WITH pwgen OR SIMILAR TOOL---
PGPASSWORD: ---PostgresSQL db password---
# MYSQL_PWD:
# MONGODB_PASSWORD:
persistence:
# -- Configure persistence settings for the chart under this key.
# @default -- See [values.yaml](./values.yaml)
data:
enabled: false
retain: true
# storageClass: ""
# accessMode: ReadWriteOnce
# size: 1Gi
subPath:
- path: borg-repository
mountPath: /mnt/borg-repository
- path: config
mountPath: /root/.config/borg
- path: cache
mountPath: /root/.cache/borg
# -- Configure SSH credentials for the chart under this key.
# @default -- See [values.yaml](./values.yaml)
ssh:
name: borgmatic-ssh
enabled: true
type: configMap
mountPath: /root/.ssh/
readOnly: false
defaultMode: 0600
configMaps:
# -- Configure Borgmatic container under this key.
# @default -- See [values.yaml](./values.yaml)
ssh :
enabled: true
data:
id_ed25519: |
-----BEGIN OPENSSH PRIVATE KEY-----
--- PASTE YOUR PRIVATE KEY HERE---
-----END OPENSSH PRIVATE KEY-----
known_hosts: |
--- paste the output of ssh-keyscan borg-repository-address ---
config:
enabled: true
data:
# -- Crontab
crontab.txt: |-
0 1 * * * PATH=$PATH:/usr/bin /usr/local/bin/borgmatic --stats -v 0 2>&1
# -- Borgmatic config. [[ref]](https://torsion.org/borgmatic/docs/reference/configuration)
# @default -- See [values.yaml](./values.yaml)
config.yaml: |
location:
# List of source directories to backup.
source_directories:
- /etc/ <- any directory you want as we are concerned only with db backup
# Paths of local or remote repositories to backup to.
repositories:
- ---BORG REPOSITORY URL---
retention:
# Retention policy for how many backups to keep.
keep_daily: 7
keep_weekly: 4
keep_monthly: 6
consistency:
# List of checks to run to validate your backups.
checks:
- name: repository
- name: archives
frequency: 2 weeks
hooks:
# Databases to dump and include in backups.
postgresql_databases:
- name: mastodon_production
hostname: hostname-of-mastodon-postgresql-db
username: mastodon
# Third-party services to notify you if backups aren't happening.
healthchecks: --- healtcheck url ---
helm install borgmatic gabe565/borgmatic -f values.yaml -n mastodon
kubectl rollout status deployment.apps/borgmatic -n mastodon
kubectl exec -i deployment.apps/borgmatic -n mastodon -- borgmatic init --encryption repokey-blake2
kubectl exec -it deployment.apps/borgmatic -n mastodon -- borgmatic create --stats
All of these commands should be completed without errors
Commen in the Fediverse @saint@river.group.lt