Object Storage on Tawa

The Short Version

Declare a bucket in catalog-info.yaml (catalog 0.3.0+). The builder provisions a MinIO bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic, short-lived, scoped to your bucket. Use the @tawa/storage SDK.

metadata:
  annotations:
    insureco.io/catalog-version: "0.3.0"
spec:
  storage:
    - name: uploads      # → S3_UPLOADS_BUCKET (+ S3_BUCKET alias if single-bucket)
      tier: s3-md        # s3-sm | s3-md | s3-lg | s3-xl

import { createStorage } from '@tawa/storage'
const storage = createStorage()                 // reads S3_* env vars (single bucket)
await storage.upload('report.pdf', buffer)

Tiers

How credentials work (and why that matters)

Credentials are dynamic MinIO users minted by a custom Vault secrets engine. Pods never hold a static key:

1. The Vault Agent sidecar reads minio/creds/{role} → the plugin creates an ephemeral MinIO user scoped to your bucket, with a lease (default_ttl: 1h, max_ttl: 24h).
2. The creds are written to /vault/secrets/storage and sourced into process.env.
3. When the lease rolls, the sidecar re-renders and restarts the container with fresh creds.

This is self-consistent: Vault always creates the MinIO user at the same moment it issues the credential, so MinIO, Vault, and the pod stay in sync.

MinIO IAM reset recovery (self-heal)

The failure it guards against

The roll above only fires on Vault's lease clock. An out-of-band MinIO IAM reset — a reinstall, a .minio.sys wipe, or a volume loss — is invisible to Vault: MinIO loses every Vault-created user, but Vault still holds the leases and believes they're valid, so no roll fires. Every storage pod keeps serving cached creds that now throw InvalidAccessKeyId, silently, until max_ttl.

You cannot fix this inside the roll: the roll is time-driven and the reset has no signal into Vault. The builder runs a storage reconciler that supplies that signal from outside.

How the reconciler works

Every STORAGE_RECONCILE_INTERVAL_MS (default 2 min) the builder:

1. Vault gate — skips the pass entirely if Vault is unhealthy (an untrustworthy probe must never trigger a heal).
2. Admin probe — checks MinIO is reachable with root creds. If MinIO is down, that's an outage, not a reset → alert only, never heal.
3. Canary probe — holds a dedicated dynamic-cred "canary" identity and exercises the exact path pods use. If the canary's creds are rejected as a non-existent user (InvalidAccessKeyId / SignatureDoesNotMatch) while admin still works, MinIO has forgotten an account Vault issued — an IAM reset.
4. Debounce — requires STORAGE_CANARY_FAILURE_THRESHOLD (default 3) consecutive failures before acting, so a transient blip never restarts the fleet.

On a confirmed reset (and outside STORAGE_HEAL_COOLDOWN_MS), it heals every storage service: re-asserts the Vault role, recreates buckets if also wiped, revokes the stale Vault leases, and kubectl rollout restarts the deployment. On restart the Vault Agent re-reads creds, the plugin re-creates the MinIO user, and the pod recovers. A Septor storage.iam_reset_healed event and an alert email are emitted. The number of services healed per run is capped by STORAGE_HEAL_MAX_SERVICES (default 50).

Non-destructive: the heal only touches the identity layer (Vault leases, MinIO users, pod lifecycle). Object data is never deleted. If a reset also wiped object data, buckets are recreated empty — the reconciler restores access, not lost objects. Object durability is MinIO's job (distributed/erasure-coded mode + backups).

Manual / on-demand heal

When ops knowingly reset MinIO, freshly-minted canary creds would mask the drift, so detection is bypassed — call the platform-admin endpoint to re-provision the whole fleet immediately:

POST /storage/reconcile   { "reason": "minio reinstall 2026-06-24" }   # platform-admin
GET  /storage/health                                                   # last reconciler result

Configuration

Roadmap

The durable long-term fix is to federate storage identity (MinIO OIDC backed by Bio-ID) so there is no stored MinIO user to lose at all — the same model AWS IRSA / GCP Workload Identity use. The reconciler remains the right safety net even after that lands.

Common Mistakes

• Expecting a stale storage pod to recover on its own after a MinIO reset — it won't until the lease rolls; the reconciler is what forces it (or POST /storage/reconcile).
• Assuming the heal can recover lost objects — it restores access, not data.
• Declaring spec.storage below catalog 0.3.0 — the field is ignored.