Database Backups on Tawa

Koko first. Use koko for every read and write on a service database. Back up before any prod write. See Koko Data Access.

The Short Version

Every MongoDB and PostgreSQL database on the platform is backed up automatically every night, and you can take an on-demand backup of your own service's database any time. Backups are stored in platform object storage (MinIO) with grandfather-father-son retention. Nothing to declare in catalog-info.yaml — backups are on by default.

koko backup -s my-api -e prod        # take a backup now
koko backups -s my-api -e prod       # list backups
koko backup-download <id> -s my-api # get a temporary download link
koko restore <id> -s my-api --confirm   # restore (admin, destructive)

A service with a Postgres database (spec.databases: [{ type: postgres }]) is backed up the same way. On-demand, pass "type": "postgres" in the request body (see Postgres databases); a service that declares only Postgres gets it by default.

koko backup starts a backup and returns straight away (202) with its id — the dump itself runs for minutes on a large database. Poll koko backups until that id shows completed before you rely on it.

Automatic Nightly Backups

The builder runs a platform-wide backup every night (02:00 UTC by default). For each MongoDB database it runs mongodump --archive --gzip, and for each PostgreSQL database pg_dump -Fc, and uploads the archive to the tawa-db-backups bucket. An engine that is not configured on the builder (no admin URI) is skipped; a failure to list one engine's databases counts as one failed backup and never stops the other engine.

Each nightly run is tagged with a retention tier (grandfather-father-son):

Older automated backups beyond the keep count are pruned automatically. Manual backups are never auto-pruned — you delete those yourself.

On-Demand (Self-Service) Backups

Take a backup of your own service's database whenever you want — before a risky migration, a data fix, or a release.

koko backup -s my-api -e prod

This dumps your service's MongoDB database, uploads it, and records it. Requires the member org role on the service's org.

It returns immediately — the backup runs in the background

A mongodump of a few hundred MB takes minutes. The request does not wait for it. The builder writes the backup record, replies 202 Accepted with the id, and runs the dump detached:

// POST /services/my-api/databases/backup  ->  202
{ "success": true,
  "data": { "backupId": "350924a4-…", "status": "running",
            "databaseName": "my-api-prod", "environment": "prod",
            "startedAt": "2026-09-16T01:51:16.845Z" } }

A 202 does not mean the backup is done. Poll the id until it leaves running:

koko backups -s my-api -e prod          # the running row is listed, with its status

GET /services/my-api/databases/backups/350924a4-…
  -> { "status": "running"   }   # still dumping
  -> { "status": "completed", "sizeBytes": 782417699, "completedAt": "…" }
  -> { "status": "failed",    "error": "mongodump failed for …" }

Only a completed backup can be downloaded or restored — the download and restore endpoints reject anything else with 409 NOT_RESTORABLE.

Why not just wait? The builder sits behind a reverse proxy that severs an idle upstream connection at 60 seconds. Holding the request open for the whole dump meant every database over roughly a few hundred MB reported 504 Gateway Timeout for a backup that in fact completed normally — right at the moment an operator was about to do something risky. See iec-builder#124.

One backup per database at a time

Asking for a second backup of a database that already has one running is refused with 409 BACKUP_IN_PROGRESS, and the error carries the in-flight id so you can poll that one instead:

{ "success": false,
  "error": { "code": "BACKUP_IN_PROGRESS",
             "message": "A backup of \"my-api-prod\" is already running (id 350924a4-…, started …)",
             "details": { "backupId": "350924a4-…", "startedAt": "…" } } }

Treat a 409 as "it is already happening", not as a failure. Two concurrent dumps of one database double the load for no benefit.

Waiting synchronously (wait=true)

?wait=true — or {"wait": true} in the body — restores the old behaviour and returns the finished record with 200:

POST /services/my-api/databases/backup?wait=true

This is subject to the proxy's read timeout (60s by default). It is fine for a small sandbox database and a bad idea for anything large — that timeout is exactly the problem 202 exists to avoid, and no client-side flag can raise it. Prefer the 202 + poll path.

Listing

koko backups -s my-api -e prod

Shows both your manual backups and the nightly automated ones for your database, newest first, with size, tier, who took it, and status. In-flight backups appear here too, as running, from the moment they start — both on-demand and nightly ones. A dump that dies leaves a durable failed row with the error, so "nothing in the list" now genuinely means "no backup was ever attempted".

Downloading

koko backup-download <backup-id> -s my-api -e prod

Returns a short-lived (15-minute) presigned URL to the gzipped archive. The archive is a standard mongodump --archive --gzip file — restore it anywhere with mongorestore --archive=<file> --gzip.

Postgres databases

Postgres backups (iec-builder#233) use the same endpoints, records, bucket, key scheme and retention as MongoDB. What differs:

Choosing the database for an on-demand backup:

// POST /services/my-api/databases/backup
{ "environment": "prod", "type": "postgres" }   // type: "mongodb" | "postgres"

Omit type and you get the MongoDB database, unless the service declares only Postgres. A service with both has two databases of the same default name (my-api-prod), so the engine is part of everything that identifies a backup: the in-flight check (409 BACKUP_IN_PROGRESS is per engine), retention groups, and the object key suffix.

A Postgres restore runs in one transaction, so a restore that fails part-way rolls back and leaves the database as it was. Download gives a standard custom-format archive — restore it anywhere with pg_restore --clean --if-exists -d <db> <file>.

The dump runs as the builder's admin role (DB_POSTGRES_ADMIN_URI). Its password is passed in PGPASSWORD, never on the command line.

koko backup does not have a --type flag yet. Until it does, call the endpoint directly for a Postgres on-demand backup; the nightly run covers Postgres either way.

Restore

Restore is destructive: for MongoDB it runs mongorestore --drop, which drops the target collections and replaces them from the backup, scoped to the original database only (--nsInclude). For Postgres it runs pg_restore --clean against the original database only. No other database is ever touched.

koko restore <backup-id> -s my-api --confirm

Requires the admin org role and the explicit --confirm flag. Without --confirm the request is rejected.

API Endpoints

Self-service backups are exposed by the builder, mirroring the tawa db connect whitelist model:

Where Backups Live

Backups are stored in the platform MinIO object store, bucket tawa-db-backups, keyed:

{env}/{service}/{tier}/{db}-{timestamp}.archive.gz   # MongoDB
{env}/{service}/{tier}/{db}-{timestamp}.pgdump       # PostgreSQL

On disk

MinIO's data is on the attached DO volume (volume-sfo3-01, ext4, /mnt/volume_sfo3_01) since 2026-10-03, at /mnt/volume_sfo3_01/minio, bind-mounted at /data/minio. Before that it was a plain directory on the builder host's root disk, where tawa-db-backups alone (~79 GB) had pushed the disk to 87-88% full (iec-builder#231).

The path MinIO is given must stay /data/minio. /etc/default/minio keeps MINIO_VOLUMES="/data/minio" and the unit's ExecStart is untouched: .minio.sys records the drive endpoint it was formatted with, so pointing MinIO at /mnt/volume_sfo3_01/minio directly (a ExecStart or MINIO_VOLUMES override) fails at startup with file access denied endpoint="/data/minio" / 0 drives provided, and /minio/health/cluster/read returns 503. A bind mount changes where the bytes live without changing the path MinIO sees.

Three pieces make it boot-safe:

The drop-in carries no ExecStart override. An earlier version of this page described an override.conf that repointed MinIO at the volume; that approach does not work (above).

Verifying it

findmnt /data/minio                                   # SOURCE /dev/sda[/minio]
for e in live cluster cluster/read; do
  curl -s -o /dev/null -w "$e %{http_code}\n" http://127.0.0.1:9000/minio/health/$e; done   # all 200
journalctl -u minio | grep -iE 'denied|0 drives'      # nothing
systemd-analyze verify /etc/systemd/system/minio.service ; findmnt --verify

/minio/health/cluster/read is the one that caught the failed path change: live returns 200 even when no drive is usable. Also do a real put and get (make a throwaway bucket, upload a file, download it, compare checksums, delete both).

Migrating or repeating the move

1. Pre-sync while MinIO runs: rsync -aHAX --delete /data/minio/ <volume>/minio/.
2. systemctl stop minio; final rsync -aHAX --delete and compare file counts and total bytes exactly. A --checksum pass over ~86 GB took about 10 minutes with MinIO down; a size+mtime pass after the pre-sync is much shorter.
3. mv /data/minio /data/minio.old-<YYYYMMDD-HHMM>, mkdir /data/minio, chown tawa:tawa.
4. Back up /etc/fstab, add the line above, mount /data/minio, add the drop-in, systemctl daemon-reload, systemctl start minio, then run the checks above.

Rollback: systemctl stop minio, umount /data/minio, rmdir /data/minio, move /data/minio.old-* back to /data/minio, delete the drop-in and the fstab line, daemon-reload, start, re-check health.

Deleting the old copy

Keep /data/minio.old-* (it is the rollback and ~86 GB of root disk) until both the next nightly backup (02:00 UTC) and the off-site mirror (03:30 UTC) have completed against the new location. Only then delete it; root-disk usage drops to about 13%. The off-site copy is mc mirrored from MinIO, so it is only a valid check once it has run after the move.

Backup Record Status

Every backup — on-demand and nightly — is written the moment its dump starts and updated when it finishes:

A running row is not a stuck row — a multi-GB database legitimately takes many minutes. BACKUP_DUMP_TIMEOUT_MS (default 120 minutes) is the ceiling after which the dump is killed and the record moves to failed.

Off-Site Replication (Disaster Recovery)

The platform MinIO runs on the same host as MongoDB, so same-box backups protect against bad migrations and accidental data loss — not against total host loss. To cover that, the backup bucket is replicated nightly to an off-site MinIO on a separate DigitalOcean droplet (tawa-db-backups, region sfo2 — a different region from the builder host in sfo3).

02:00 UTC  builder dumps all DBs        -> tawa MinIO  (tawa-db-backups, on the builder host)
03:30 UTC  systemd timer: mc mirror     -> off-site MinIO droplet (sfo2, 200GB volume)

• Replication is an incremental mc mirror --overwrite --remove, so the off-site copy tracks retention pruning.
• Script: iec-builder/scripts/backup-offsite-sync.sh (reads /etc/tawa-backup-offsite.env); units: tawa-backup-offsite.{service,timer}.
• The off-site droplet's MinIO port is firewalled to the builder host's IP only.

If the builder host is lost entirely, the latest backups still exist on the off-site droplet and can be restored from there.

Configuration (Platform Operators)

Controlled by builder environment variables (defaults shown):

Backups require MinIO admin credentials in Vault KV (secret/minio/admin) plus an admin URI per engine: DB_MONGODB_ADMIN_URI for MongoDB, DB_POSTGRES_ADMIN_URI for PostgreSQL (pg_dump / pg_restore 16 on the builder host's PATH). An engine without its URI is skipped; if neither is set, the scheduler logs a warning and stays idle.

What NOT to Do

# ❌ WRONG: treating the 202 as "the backup is done"
#    202 means "started". Poll the id until status leaves "running" before you
#    migrate — a backup you have not confirmed is not a safety net.

# ❌ WRONG: re-running koko backup because the first one "didn't come back"
#    It did come back, with 202. A second request is refused with 409
#    BACKUP_IN_PROGRESS anyway; poll the id the 409 hands you.

# ❌ WRONG: reaching for ?wait=true on a large database
#    It re-creates the 60s proxy timeout the 202 path exists to avoid.

# ❌ WRONG: restore without --confirm — rejected (destructive guard)
koko restore <id> -s my-api

# ❌ WRONG: assuming same-box MinIO is full disaster recovery
#    It protects against bad data, not host loss. Off-box replication is separate.

# ✅ CORRECT: snapshot before a risky migration, then restore if it goes wrong
koko backup -s my-api -e prod           # 202 — note the backupId
koko backups -s my-api -e prod          # poll until that id shows "completed"
# ... run migration ...
koko restore <id> -s my-api --confirm   # only if needed