Object Storage on Tawa

The Short Version

Declare a bucket in catalog-info.yaml (catalog 0.3.0+). The builder provisions a MinIO bucket with a hard quota, and credentials are injected into your pod by Vault — dynamic, short-lived, scoped to your bucket. Use the @insureco/storage SDK at ^0.2.0.

metadata:
  annotations:
    insureco.io/catalog-version: "0.3.0"
spec:
  storage:
    - name: uploads      # → S3_UPLOADS_BUCKET (+ S3_BUCKET alias if single-bucket)
      tier: s3-md        # s3-sm | s3-md | s3-lg | s3-xl

import { createStorage } from '@insureco/storage'   // ^0.2.0 — see credentials section
const storage = createStorage()                     // single bucket
await storage.upload('report.pdf', buffer)

The package is @insureco/storage. A @tawa/storage exists in the registry but is abandoned at 0.1.0 and reads credentials from process.env, which is broken in a deployed pod (see below). Do not use it.

Tiers

How credentials work (and why that matters)

Credentials are dynamic MinIO users minted by a custom Vault secrets engine. Pods never hold a static key:

1. The Vault Agent sidecar reads minio/creds/{role} → the plugin creates an ephemeral MinIO user scoped to your bucket, with a lease (default_ttl: 1h, max_ttl: 24h).
2. The creds are written to /vault/secrets/storage, which the sidecar keeps current.
3. @insureco/storage ≥ 0.2.0 reads that file — not process.env — re-reading it before each operation, so a rolled lease is picked up with no restart.

NEVER read S3_* out of process.env

Requires "@insureco/storage": "^0.2.0". A ^0.1.x pin cannot resolve it and will read process.env, which is broken. Check yours before assuming storage works.

The pod's S3_* environment variables are a snapshot taken at boot and they go stale within about an hour. There are two Vault Agent renders, not one:

• the init container renders the file, the entrypoint sources it into the environment and exec()s your process — so your env holds that credential, forever;
• the sidecar then renders the file again, getting a different credential, and renews only its own.

Nothing renews the init container's credential. When its Vault token expires (~1h) the credential it created is revoked, and every S3_* value in your environment is dead — while the file on disk is still valid. Measured across production: every Vault-using pod is in this state, and pods older than about an hour fail InvalidAccessKeyId on the env credential.

So:

import { createStorage, hasVaultFile } from '@insureco/storage'

const storage = createStorage()          // ✅ reads the Vault file when deployed
const storage2 = createStorage('uploads') // ✅ named bucket, same behaviour

// ❌ never do this — these values are a dead boot snapshot in a deployed pod
new S3Client({ credentials: { accessKeyId: process.env.S3_ACCESS_KEY_ID, ... } })

// ❌ and never gate on them either — they are present whether or not they work
if (process.env.S3_ACCESS_KEY_ID) { /* proves nothing */ }
if (hasVaultFile()) { /* ✅ deployed with Vault-managed creds */ }

hasVaultFile() is the correct "am I deployed?" discriminator: file when deployed, env when local. Tracked in iec-builder #31; see also iec-docman #7 for what it looks like when it bites.

Rotation handling (fixed — iec-builder #31)

Historically broken, now fixed. The builder used to ask the Vault Agent to rm -f /vault/signals/alive on rotation and probe cat on that file. It never worked for any service: the command runs in the vault-agent sidecar, which never had the vault-signals volume mounted, so it removed nothing — and rm -f on a missing path exits 0, so it reported success while doing nothing.

There is also no longer a Vault init container (agent-pre-populate: false). Two agent processes meant two independent reads, hence two credentials — the app captured one nobody renewed. One render now means the credential in your environment is the one being kept alive.

Rotation is detected inside your own container instead: the entrypoint fingerprints the database credentials it sourced, and the liveness probe re-fingerprints the live files. A mismatch means Vault issued credentials the running process cannot pick up (its environment is fixed after exec), so Kubernetes restarts it. The probe fails open — missing fingerprint, missing files, or no md5sum all pass, so it can never cause a restart loop.

Storage is deliberately excluded from that fingerprint: @insureco/storage re-reads the file per operation, so a storage roll needs no restart and must not cause one.

MinIO IAM reset recovery (self-heal)

The failure it guards against

The roll above only fires on Vault's lease clock. An out-of-band MinIO IAM reset — a reinstall, a .minio.sys wipe, or a volume loss — is invisible to Vault: MinIO loses every Vault-created user, but Vault still holds the leases and believes they're valid, so no roll fires. Every storage pod keeps serving cached creds that now throw InvalidAccessKeyId, silently, until max_ttl.

You cannot fix this inside the roll: the roll is time-driven and the reset has no signal into Vault. The builder runs a storage reconciler that supplies that signal from outside.

How the reconciler works

Every STORAGE_RECONCILE_INTERVAL_MS (default 2 min) the builder:

1. Vault gate — skips the pass entirely if Vault is unhealthy (an untrustworthy probe must never trigger a heal).
2. Admin probe — checks MinIO is reachable with root creds. If MinIO is down, that's an outage, not a reset → alert only, never heal.
3. Canary probe — holds a dedicated dynamic-cred "canary" identity and exercises the exact path pods use. If the canary's creds are rejected as a non-existent user (InvalidAccessKeyId / SignatureDoesNotMatch) while admin still works, MinIO has forgotten an account Vault issued — an IAM reset.
4. Debounce — requires STORAGE_CANARY_FAILURE_THRESHOLD (default 3) consecutive failures before acting, so a transient blip never restarts the fleet.

On a confirmed reset (and outside STORAGE_HEAL_COOLDOWN_MS), it heals every storage service: re-asserts the Vault role, recreates buckets if also wiped, revokes the stale Vault leases, and kubectl rollout restarts the deployment. On restart the Vault Agent re-reads creds, the plugin re-creates the MinIO user, and the pod recovers. A Septor storage.iam_reset_healed event and an alert email are emitted. The number of services healed per run is capped by STORAGE_HEAL_MAX_SERVICES (default 50).

Non-destructive: the heal only touches the identity layer (Vault leases, MinIO users, pod lifecycle). Object data is never deleted. If a reset also wiped object data, buckets are recreated empty — the reconciler restores access, not lost objects. Object durability is MinIO's job (distributed/erasure-coded mode + backups).

Manual / on-demand heal

When ops knowingly reset MinIO, freshly-minted canary creds would mask the drift, so detection is bypassed — call the platform-admin endpoint to re-provision the whole fleet immediately:

POST /storage/reconcile   { "reason": "minio reinstall 2026-06-24" }   # platform-admin
GET  /storage/health                                                   # last reconciler result

Configuration

Roadmap

The durable long-term fix is to federate storage identity (MinIO OIDC backed by Bio-ID) so there is no stored MinIO user to lose at all — the same model AWS IRSA / GCP Workload Identity use. The reconciler remains the right safety net even after that lands.

Common Mistakes

• Expecting a stale storage pod to recover on its own after a MinIO reset — it won't until the lease rolls; the reconciler is what forces it (or POST /storage/reconcile).
• Assuming the heal can recover lost objects — it restores access, not data.
• Declaring spec.storage below catalog 0.3.0 — the field is ignored.