Database Backups on Tawa

The Short Version

Every MongoDB database on the platform is backed up automatically every night, and you can take an on-demand backup of your own service's database any time. Backups are stored in platform object storage (MinIO) with grandfather-father-son retention. Nothing to declare in catalog-info.yaml — backups are on by default.

koko backup -s my-api -e prod        # take a backup now
koko backups -s my-api -e prod       # list backups
koko backup-download <id> -s my-api # get a temporary download link
koko restore <id> -s my-api --confirm   # restore (admin, destructive)

Automatic Nightly Backups

The builder runs a platform-wide backup every night (02:00 UTC by default). For each database it runs mongodump --archive --gzip and uploads the archive to the tawa-db-backups bucket.

Each nightly run is tagged with a retention tier (grandfather-father-son):

Older automated backups beyond the keep count are pruned automatically. Manual backups are never auto-pruned — you delete those yourself.

On-Demand (Self-Service) Backups

Take a backup of your own service's database whenever you want — before a risky migration, a data fix, or a release.

koko backup -s my-api -e prod

This dumps your service's MongoDB database, uploads it, and records it. Requires the member org role on the service's org.

Listing

koko backups -s my-api -e prod

Shows both your manual backups and the nightly automated ones for your database, newest first, with size, tier, who took it, and status.

Downloading

koko backup-download <backup-id> -s my-api -e prod

Returns a short-lived (15-minute) presigned URL to the gzipped archive. The archive is a standard mongodump --archive --gzip file — restore it anywhere with mongorestore --archive=<file> --gzip.

Restore

Restore is destructive: it runs mongorestore --drop, which drops the target collections and replaces them from the backup. It is scoped to the original database only (--nsInclude), so no other database is ever touched.

koko restore <backup-id> -s my-api --confirm

Requires the admin org role and the explicit --confirm flag. Without --confirm the request is rejected.

API Endpoints

Self-service backups are exposed by the builder, mirroring the tawa db connect whitelist model:

Where Backups Live

Backups are stored in the platform MinIO object store, bucket tawa-db-backups, keyed:

{env}/{service}/{tier}/{db}-{timestamp}.archive.gz

Off-Site Replication (Disaster Recovery)

The platform MinIO runs on the same host as MongoDB, so same-box backups protect against bad migrations and accidental data loss — not against total host loss. To cover that, the backup bucket is replicated nightly to an off-site MinIO on a separate DigitalOcean droplet (tawa-db-backups, region sfo2 — a different region from the builder host in sfo3).

02:00 UTC  builder dumps all DBs        -> tawa MinIO  (tawa-db-backups, on the builder host)
03:30 UTC  systemd timer: mc mirror     -> off-site MinIO droplet (sfo2, 200GB volume)

• Replication is an incremental mc mirror --overwrite --remove, so the off-site copy tracks retention pruning.
• Script: iec-builder/scripts/backup-offsite-sync.sh (reads /etc/tawa-backup-offsite.env); units: tawa-backup-offsite.{service,timer}.
• The off-site droplet's MinIO port is firewalled to the builder host's IP only.

If the builder host is lost entirely, the latest backups still exist on the off-site droplet and can be restored from there.

Configuration (Platform Operators)

Controlled by builder environment variables (defaults shown):

Backups require DB_MONGODB_ADMIN_URI (admin Mongo access) and MinIO admin credentials in Vault KV (secret/minio/admin). If DB_MONGODB_ADMIN_URI is unset, the scheduler logs a warning and stays idle.

What NOT to Do

# ❌ WRONG: restore without --confirm — rejected (destructive guard)
koko restore <id> -s my-api

# ❌ WRONG: assuming same-box MinIO is full disaster recovery
#    It protects against bad data, not host loss. Off-box replication is separate.

# ✅ CORRECT: snapshot before a risky migration, then restore if it goes wrong
koko backup -s my-api -e prod
# ... run migration ...
koko restore <id> -s my-api --confirm   # only if needed