4 Backup
Anton Nesterov edited this page 2026-11-08 21:58:35 +01:00

Backup

The recommended backup path never touches the production master: run a replica (see Replication), stop it, snapshot its data directory, start it again. The replica catches up from the master's changeset log the moment it returns; production serves traffic without a pause, and the snapshot is a complete, consistent copy (login state, schema, files) that can be restored as a standalone server.

Back up from a replica

  1. Run at least one replica; a dedicated one that serves no client traffic is ideal for backups.
  2. On your backup schedule, stop the replica (docker stop, docker compose stop, or kubectl scale) and wait for it to exit. A clean stop checkpoints the SQLite WAL into the database files.
  3. Snapshot its DATA_DIR: tar, rsync, or a volume snapshot.
  4. Start the replica again. It resyncs incrementally from the master's changeset log; nothing is lost, nothing pauses on the master.

Keep the stop shorter than REPLICATION_LOG_RETENTION_HOURS (default 24) so the replica resumes incrementally; a longer outage just means one full re-download from the master when it comes back, which is still safe.

Example: nightly cron with docker

# /usr/local/bin/vski-backup.sh
#!/bin/bash
set -euo pipefail

BACKUP_DIR="/var/lib/vski/backups"
DATE=$(date +%Y-%m-%d_%H%M%S)

# stop the backup replica; a clean exit checkpoints the WAL
docker compose -f /srv/vski/docker-compose.yml stop vski-backup

# snapshot its volume (name: <compose project>_<volume>; the hosting guide's
# stack is project "vski" with volume "vski-backup")
tar -C "$(docker volume inspect vski_vski-backup --format '{{ .Mountpoint }}')" \
    -czf "$BACKUP_DIR/$DATE.tar.gz" .

# back in business: the replica resyncs from the changeset log
docker compose -f /srv/vski/docker-compose.yml start vski-backup

# keep 30 days
find "$BACKUP_DIR" -name "*.tar.gz" -mtime +30 -delete
# crontab -e : nightly at 3am
0 3 * * * /usr/local/bin/vski-backup.sh >> /var/log/vski-backup.log 2>&1

On Kubernetes, the same procedure is a CronJob that stops the backup replica, snapshots its PVC (or kubectl cp from a mounted volume after the pod has exited) and starts it again. With the vski-cluster chart the backup node is the highest ordinal, so stopping it means scaling the StatefulSet down by one. Check first that this pod is a replica (/tango/status, or the role map ConfigMap): after a promotion it can be the master, and then promote another node before scaling down.

Restoring

Point a server at the snapshot (DATA_DIR=<extracted snapshot> in master mode) and it runs as a complete standalone instance: admin accounts, collections, and records are all there. If you restore it into a cluster, replicas treat its history as foreign and re-download from it once: a restored master is a point-in-time fork, and that is exactly what a backup restore should be.

S3 file storage

With S3_BUCKET configured, uploaded files live in the bucket, not in DATA_DIR; back the bucket with its own tooling (versioning, lifecycle rules, cross-region replication). The procedures above then cover the databases; the bucket covers the files. In clustered setups this is the recommended mode anyway; see File Sync.

Data directory access rules

The server is the only process that should ever access DATA_DIR:

  • Databases run in WAL mode; each <db>.db is accompanied by <db>.db-wal and <db>.db-shm files. The server deliberately never deletes these on connection close, and leftover -wal/-shm files after a shutdown are recovered (checkpointed into the database) automatically on the next start; they are not junk; do not prune them by hand.
  • Replicas never read the master's data directory. They sync over HTTP from the master (MASTER_URL) into their own DATA_DIR; never mount or share the master's data directory with another process, replicas included.
  • Do not point external tools (the sqlite3 CLI, file backup agents, monitoring) at the live database files. A client that opens and closes them triggers SQLite's last-connection cleanup, which can delete the live -wal/-shm out from under the server's pooled connections and poison them until restart (silently stale reads, database disk image is malformed on writes). The server holds guard locks to deter this, but the supported access paths are the HTTP API, the SQL console, and replicas, not the files.

This is why snapshots are taken from a stopped server (or a stopped replica, as above): the -wal/-shm state must never be captured mid-transaction.

rsync snapshots (single server)

Without a replica (one server, no cluster) the same idea applies to the server itself: stop it, snapshot, start it. Use rsync with --link-dest to keep the snapshots space-efficient:

# Create a backup script: /usr/local/bin/vski-backup.sh
#!/bin/bash

DATA_DIR="/var/lib/vski/data"
BACKUP_DIR="/var/lib/vski/backups"
DATE=$(date +%Y-%m-%d_%H%M%S)
LATEST="$BACKUP_DIR/latest"

mkdir -p "$BACKUP_DIR"

# stop the server: snapshots must never race a live WAL
docker stop vski

# Create incremental snapshot with hard links for unchanged files
rsync -av --delete \
  --link-dest="$LATEST" \
  "$DATA_DIR/" \
  "$BACKUP_DIR/$DATE/"

# restart; if this node has replicas or a changeset log, it resumes where it left off
docker start vski

# Update latest symlink
ln -sfn "$BACKUP_DIR/$DATE" "$LATEST"

# Remove backups older than 30 days
find "$BACKUP_DIR" -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;

echo "Backup completed: $BACKUP_DIR/$DATE"

This approach creates full backup directories but uses hard links for unchanged files, so each snapshot appears complete while only storing differences. For a server that must not pause, add a replica and use the procedure above instead.