Build & Deploy Pipeline

The iec-builder executes a fixed, ordered pipeline for every tawa deploy. Understanding each step — what it does, when it fails hard vs. fails open, and what it writes to each system — is essential for debugging deploys and hardening the platform.

Pipeline Overview

tawa deploy --prod
     │
     ▼
 iec-builder (builder.ts: executeBuild)
     │
     ├─ PRE-BUILD
     │   ├─ Clone/extract
     │   ├─ Catalog parse
     │   ├─ Dockerfile strategy
     │   ├─ Deploy gate (wallet reserve)
     │   ├─ Config preflight
     │   └─ Convention check / SDK upgrade gate
     │
     ├─ BUILD
     │   ├─ docker build
     │   └─ docker push → registry
     │
     └─ DEPLOY  (ENABLE_K8S_DEPLOY=true)
         ├─ Namespace + Koko namespace registration
         ├─ Database provisioning (Vault or static K8s secrets)
         ├─ Consumed database provisioning
         ├─ Object storage provisioning (Vault, catalog 0.3.0+)
         ├─ Dependency resolution (Koko + Bio-ID scope grants)
         ├─ OAuth client provisioning
         ├─ Claude access: Anthropic identity + ids (when switched on)
         ├─ Scope grant creation
         ├─ Module registration (Bio-ID)
         ├─ userAccess provisioning
         ├─ helm upgrade --install
         ├─ Deploy snapshot (Koko, fire-and-forget)
         ├─ DNS configuration (Cloudflare)
         ├─ Post-deploy pod health check
         ├─ Vault secret readback (gates re-rolls)
         ├─ Deploy-gated tests (iec-test, catalog 0.5.0+)
         ├─ Service registration in Koko
         ├─ Queue + schedule binding registration + prune
         ├─ Role + registration metadata (catalog 0.6.0+)
         └─ Uptime monitor (iec-pulse, fire-and-forget)

Status Flow

queued → cloning → building → pushing → deploying → testing → completed
                                                            ↘ failed (any step)

Two triggers, one build. A tawa deploy for a service + environment whose branch tip is already being built does not queue a second build: the trigger answers 202 with the in-flight build's id and deduplicated: true, and tawa deploy --watch attaches to it. An in-flight build that has not yet cloned always counts as a build of the tip; one that has counts only if its commit is still the branch head (one git ls-remote). If main moved in between, you get a real new build. The in-flight build's log records who attached (iec-builder#110).

First Deploy: the deploy → config → deploy ordering

Nothing can be attached to a service until the service record exists — and on a new service, the first tawa deploy is what creates it.

Every service-keyed thing — config vars, managed secrets, database credentials, whitelist entries, authorized orgs — hangs off a record in the builder's services collection. Until that record exists, the routes that write those things have nothing to find, and 404.

tawa deploy --prod           # 1. creates the service record, then builds
tawa config set KEY=value    # 2. now this works
tawa deploy --prod           # 3. redeploy — this is what puts the value in the pod

Full config reference, including tawa config set --secret KEY --from-locker lk://insureco/<vault>/<item> for a secret whose plaintext never passes through a terminal: config.

Step 3 is not optional. tawa config set writes to the builder's config store, not to the running pod. Config and secrets are materialized into Kubernetes at deploy time (steps 4b and 18), so a value set after a deploy reaches the pod only on the next deploy.

Run it in the wrong order and the builder's config routes answer — not the CLI:

Service 'my-new-api' not found. A service record is created by its first deploy,
so config cannot be set before then. …

That is not a typo and not a permissions problem: the service has simply never been deployed.

How the record actually gets created

The record is written by POST /services (servicesRouter.post('/')), and two things call it:

Note the build trigger itself does not create anything: POST /builds/trigger 404s with Service not found if the record is missing. Registration strictly precedes every build.

So tawa link is the escape hatch when you want config in place before the first build ever runs — it creates the record without spending a build:

tawa link                    # creates the service record, no build
tawa config set KEY=value    # works immediately
tawa deploy --prod           # first build already has its config

It compounds on a first deploy

A new service with a scoped spec.dependencies entry can fail the scope gate (steps 13/15), and then — when you go to set the config it needs — fail again on config-before-deploy. Two unrelated-looking failures, one cause: nothing exists yet. A first deploy that goes in this order hits neither:

1. tawa preflight — catch catalog problems before spending a build.
2. tawa link — create the service record (or let the first tawa deploy do it).
3. tawa config set … — now the service exists, so this works.
4. tawa deploy --prod — may still block on a scope grant; the record already exists either way.
5. Approve any blocked grants — the block message names the full grant ID and the exact command (tawa scopes grant <grantId>). A scope grant is a deploy-time gate: approving it is what unblocks the build and gets {SERVICE}_URL injected. It is not a per-request runtime check.
6. tawa deploy --prod — green.

Chicken-and-egg with configDeclarations. If the catalog declares a required config var (catalog 0.5.0+), the config preflight (step 6) hard-fails a first deploy with no value set — and you cannot set one before the record exists. Either tawa link first (above), or let the first deploy fail: registration happens before the build is queued, so the record survives and tawa config set then works. Declaring a default: avoids the round trip entirely.

Step-by-Step Reference

1. Clone / Extract

What: Clones the repo at the specified commitSha using the branch as a hint. If a tarball was uploaded (via POST /builds/upload), it extracts that instead.

Auth: Forgejo repos get a PAT injected (forgejo-token:TOKEN@host). GitHub SSH URLs are converted to HTTPS with a GitHub PAT. Both are handled by injectGitCredentials().

Output: Actual commit SHA is resolved and backfilled to the build record. Workspace at {WORKSPACE_DIR}/{buildId}.

Fail behavior: Hard fail — no source = no build.

2. Read .iec.yaml (monorepo config)

What: Reads .iec.yaml from the repo root (or {service.appPath}/.iec.yaml). Controls:

• appPath — subdirectory containing the service
• dockerfile — custom Dockerfile path
• buildContext — custom build context (relative to appPath)
• helmChart — custom Helm chart path

Priority for appPath: service.appPath (DB) > .iec.yaml > inferred from service.dockerfilePath.

Fail behavior: Missing .iec.yaml is fine — defaults apply.

3. Parse catalog-info.yaml

What: Reads catalog-info.yaml from {workDir}/{effectiveAppPath}. Parses all fields: framework, databases, routes, dependencies, auth, storage, schedules, queues, etc.

Catalog version:

• Current: 0.6.0 — supports modules, registration metadata, role IDs
• Minimum: 0.2.0 — old catalogs below this are rejected (when ENFORCE_MINIMUM_CATALOG_VERSION=true)
• 0.3.0 — storage buckets
• 0.4.0 — unified spec.dependencies with Bio-ID scope grants
• 0.5.0 — deploy-gated tests, spec.userAccess, config declarations
• 0.6.0 — spec.userAccess.roles, spec.registration

Framework auto-detection: If no catalog, the builder sniffs package.json for Next.js, Express, Hono, Fastify, etc. and constructs a minimal catalog.

Fail behavior: Missing catalog is a warn (not a hard fail) — deploy proceeds with defaults.

4. Dockerfile Strategy

Decision tree:

Has explicit Dockerfile?
  (iec.yaml OR service.dockerfilePath != 'Dockerfile')
  OR catalog.customDockerfile == 'true'?
    YES → useCustomDockerfile = true
           ├─ patchCustomDockerfile:
           │   ├─ Replace non-numeric USER → USER 1001 (K8s runAsNonRoot)
           │   └─ Inject Vault entrypoint wrapper (when VAULT_ENABLED=true)
           └─ File not found → fall back to auto-generate
    NO  → ENABLE_AUTO_DOCKERFILE=true?
           YES → generateDockerfile(buildContext, catalog)
           NO  → use project Dockerfile as-is

Important: Services using a custom Dockerfile must add the Vault entrypoint wrapper manually unless the builder patches it. See patchCustomDockerfile().

Next.js public/. The generated Next.js Dockerfile runs mkdir -p /app/public in the builder stage before the runner stage copies it out. public/ is optional in Next.js — create-next-app does not create it — and until 2026-08-21 the unconditional copy failed the image build for any service without one, after a full install and compile. Services no longer need a placeholder public/.gitkeep; existing ones are harmless and can be deleted whenever convenient. Custom Dockerfiles are on their own here, as with the Vault wrapper.

Static sites: content types. The generated static Dockerfile appends five mappings to nginx's /etc/nginx/mime.types — md, mjs, webmanifest, ttf, otf. Stock nginx maps none of them, so before 2026-09-17 each served as default_type application/octet-stream. .mjs was the sharp edge: a browser refuses an ES module with the wrong content type, so a static site's imports failed outright. Custom Dockerfiles are on their own here too.

If you write your own nginx config, where you put a types { ... } block decides whether it adds a mapping or destroys the rest of them:

The difference is the inheritance rule: a types directive at a deeper level redefines what it inherits, while two at the same level merge into one map. Both were checked against a running container — the server-level version really does take CSS down with it.

A block at either level also cannot re-include mime.types to compensate: that file is itself a types { ... } block, and nesting one inside another is a syntax error nginx refuses to start on (unexpected "{" in /etc/nginx/mime.types:2). The generated Dockerfile sidesteps the question by appending to mime.types itself.

Vault entrypoint wrapper — how dynamic secrets reach process.env. With Vault dynamic credentials, the Vault Agent renders each secret as a file under /vault/secrets/<name> (a block of KEY=VALUE lines — e.g. /vault/secrets/mongodb → MONGODB_URI=…, /vault/secrets/storage → S3_*). The generated ENTRYPOINT (/usr/local/bin/entrypoint.sh) sources those files into the environment and then exec "$@" — so from the app's perspective they are ordinary env vars. You never read /vault/secrets/* from application code.

The wrapper waits (bounded ~10s) for every secret file to be non-empty before sourcing, then sources only non-empty files ([ -s "$f" ]). This closes a startup race: the vault-agent sidecar starts alongside the app container and re-renders the dynamic creds at that instant, so an entrypoint that sourced immediately (old [ -f "$f" ] existence-only check) could read a half-written/truncated file and boot with MONGODB_URI/S3_* unset — leaving the app with no database. The wait loop is defined once in VAULT_ENTRYPOINT (dockerfile-generator.ts) and is used by both the generated Dockerfiles and patchCustomDockerfile(). Reference: https://tawa.insureco.io/reference/build-pipeline

Because the creds are dynamic (rotated by Vault), they cannot be pinned as static tawa config set values — the running app must read the live-rendered files via this entrypoint.

Non-root validation: After Dockerfile resolution, validateNonRootDockerfile() checks that the final USER directive is a number ≠ 0. Fails hard. Exception: static framework (nginx runs as root internally).

Package manager: The generated Dockerfile auto-detects the lockfile and installs with the matching tool — yarn.lock → yarn, package-lock.json → npm, pnpm-lock.yaml → pnpm. For pnpm, the builder runs corepack prepare pnpm@<pinned> --activate (NOT a bare corepack enable pnpm), because bare enable pulls the latest pnpm (11.x), which requires Node ≥ 22.13 (node:sqlite) and crashes on the Node 20 build image. A repo's own packageManager field in package.json still overrides the pinned default — pin one there if your service needs a specific pnpm version. See dockerfile-generator.ts (PINNED_PNPM).

4b. Build-time env injection (.env.production)

When: Immediately before docker build, if the catalog declares build-time vars via insureco.io/env-vars.

The builder materializes the declared vars from the config store (tawa config set values + managed secrets) into a .env.production file at the build-context root. npm run build (CRA, Vite, and Next.js all read .env* files at build time) then bakes them into the client bundle.

Why this step exists: tawa config set values live in the builder DB, not the repo (whose .env is normally gitignored), and docker build runs without --build-arg. Without this step, REACT_APP_* / NEXT_PUBLIC_* vars compile to their hardcoded fallback (often localhost) and the deployed static bundle calls the wrong API URL. (Symptom: a static/CRA app whose login/API calls hit the app's own host or localhost instead of the real API.)

What is written: ONLY the keys listed in insureco.io/env-vars that have a value in config. These are public-by-design (they end up in the client bundle). The full secret set is never written here — non-declared secrets stay runtime-only (injected as K8s secrets at deploy). Value resolution: service.config merged over decrypted service.secrets.

Build log line: Wrote N build-time env var(s) to .env.production: KEY1, KEY2 (names only — values are never logged).

Belt-and-suspenders: client apps can also resolve their API base at runtime from window.location.hostname so they don't depend on build-time injection at all. Recommended for static SPAs as a fallback. See writeBuildEnvFile() in builder.ts.

5. Deploy Gate

When: ENABLE_K8S_DEPLOY=true AND WALLET_URL is set AND catalog was parsed.

Skipped for platform services: iec-wallet, iec-janus, tawa-web, koko, bio-id, iec-test (or any service in DEPLOY_GATE_SKIP_SERVICES env var).

Three checks (in order):

Fail-open: If the wallet service is unreachable, the gas reserve check returns "sufficient" and the deploy proceeds.

Reserve formula:

required = (hosting_gas_per_hour × 24 × 30 × 3) + (storage_gas_per_month × 3)
shortfall = max(0, required - current_balance)

Authorized orgs (multi-org collaboration): The deploy gate keys off catalog.owner (= spec.owner = the primary org). A service can additionally grant one or more authorized orgs whose devs get full-parity access — deploy, config, secrets, DB — without being moved into the owning org. The deploy gate is unaffected:

• The authorized-org dev never changes spec.owner, so the owner tamper guard still passes.
• Gas is still billed to the primary org's wallet (deriveWalletId(catalog.owner)).
• The authorized org does not need to be in approved_orgs — the allowlist checks catalog.owner, not the deploying user.

Authorization is a per-service access check only (user.org === service.org OR user.org ∈ service.authorizedOrgs), enforced on the build trigger, build views, and all service management routes. Manage it as an admin of the owning org:

GET    /services/:name/authorized-orgs                 # list owner + authorized orgs
POST   /services/:name/authorized-orgs { orgSlug }     # add   (owning-org admin only)
DELETE /services/:name/authorized-orgs/:orgSlug        # remove (owning-org admin only)

Only owning-org admins (or super_admin / platform service) may add or remove authorized orgs — an authorized org cannot add further orgs. A build triggered by an authorized-org dev is stamped with that dev's org, but build history is resolved through the service so the owning org and every authorized org share one view.

6. Config Preflight

When: catalog.configDeclarations is non-empty (catalog 0.5.0+).

What: Each declared config var with required: true must exist in service.config or service.secrets. Missing vars → hard fail with a list of what's missing and the tawa config set command to fix it.

Defaults: Vars with default: are injected into the deploy even if not set by the user (lowest priority — user config overrides).

Ordering: tawa config set only works once the service record exists, and the value it writes reaches the pod on the next deploy — see First Deploy: the deploy → config → deploy ordering.

7. Convention Check + SDK Upgrade Gate

Convention check: checkConventions() runs Koko gate rules against the source files.

• warn violations: logged but non-blocking
• block violations: hard fail (bypass with tawa deploy --skip-convention-check --reason "...")
• Skip is always logged to Septor audit regardless

SDK upgrade gate: For each detected package version bump vs. the previous deploy manifest:

• Checks Koko for a changelog entry at {package}@{version}
• No changelog entry + Koko reachable → hard fail
• Koko unreachable → warn and proceed (fail-open)
• Bypassed if --skip-convention-check was set

"The previous deploy manifest" is the one recordDeploySnapshot stored after the last successful deploy (Step 18). This step only reads it; it never writes.

Correction (2026-09-06): this gate had never fired. The builder's manifest POST lacked commit, sdkVersions, and deployedBy — all three required by Koko — so every manifest was rejected 400 VALIDATION_ERROR, nothing was ever stored, and "the previous deploy manifest" was always empty. Because the write is fire-and-forget, no build ever warned. Fixed in iec-builder#95: the body now matches Koko's schema, and a rejection is written to the build log as [warn] Deploy snapshot was not stored in Koko: …. There is no backfill — the first deploy after the fix stores a manifest, and the gate has something to diff against from the second deploy onward.

8. Docker Build + Push

Build:

DOCKER_BUILDKIT=1 docker build -t {DOCKER_REGISTRY}/{service}:{sha7} -f {dockerfile} {buildContext}

BuildKit caches (iec-builder#171). Every generated Dockerfile starts with # syntax=docker/dockerfile:1 and uses cache mounts, which only BuildKit understands. BuildKit is the default on the builder host's Docker 29; DOCKER_BUILDKIT=1 is set explicitly so a daemon config change cannot quietly turn every mount into a parse error.

Layer caching already skips the install when package.json and the lockfile are unchanged. The npm/yarn cache mounts help a little when the lockfile does change: packages are re-extracted from the host cache instead of being downloaded.

There is no pnpm store mount, on purpose. pnpm hardlinks packages from its store into node_modules, and a hardlink cannot cross a cache-mount boundary. pnpm falls back to copying every file, and on the builder host that made a lockfile-change install 20–30s slower than fetching from the registry (measured, iec-builder#171). Do not add one to a custom Dockerfile either. The .next/cache mount lets next build reuse its compiler cache when source changes, where before every deploy compiled from scratch. None of the mounted content ends up in the image.

Custom Dockerfiles get none of this automatically. Add the same lines yourself, using the same ids so you share the host's caches:

# syntax=docker/dockerfile:1
RUN --mount=type=cache,id=npm,target=/root/.npm npm ci
RUN --mount=type=cache,id=nextjs-<your-service>,target=/app/.next/cache pnpm build

Also give a custom Dockerfile a .dockerignore (at least .git, node_modules, .next). Without one, COPY . . sends the whole clone, .git included, into the build context. Keep anything the runtime reads: bio keeps docs/** while excluding other *.md (iec-bio#134). Never exclude .env.production: Step 4b writes the build-time vars there.

Caches last only as long as the host keeps its BuildKit cache. A host-level prune (docker builder prune) empties them, and the build after that runs cold.

Context size check: Warns if build context exceeds threshold (avoids accidental large builds).

Push: docker push {imageTag} to registry.insureco.io (Forgejo's package registry; its layers live in DO Spaces nyc3). This is the canonical copy, and a failure here is a hard failure.

Mirror push (optional, off by default; iec-builder#175): when MIRROR_REGISTRY is set, the builder also pushes the same image to the in-cluster mirror tawa-mirror (insureco/tawa-cluster#2) under the same repo path: registry.insureco.io/insureco/{service}:{sha7} becomes {MIRROR_REGISTRY}/insureco/{service}:{sha7}. The builder host and the mirror share the VPC, so the copy takes seconds.

• Why: cluster nodes egress through the VPC NAT gateway at about 1 MB/s per flow, so a 226 MB image took 3-4 min per node (iec-builder#173).
• Non-blocking: any failure is [warn] Mirror push … failed … The deploy continues, and nodes pull from upstream as before.
• A misconfiguration (a scheme or path in MIRROR_REGISTRY, or missing credentials) is a [warn], and the mirror push is skipped.
• Credentials: MIRROR_REGISTRY_USER / MIRROR_REGISTRY_TOKEN (the mirror's pusher user, daryx vault tawa-mirror-pusher) go to docker login --password-stdin over stdin. They never appear in argv or a build log line.
• Until nodes carry a containerd hosts.toml entry for the mirror (planned after the DOKS 1.36 upgrade), the copy is pushed but not yet pulled from.

Fail behavior: Hard fail on any non-zero exit.

9. Namespace + Koko Namespace Registration

Namespace: kubectl create namespace {service}-{environment} (idempotent).

Koko registration: Registers namespace with a gas multiplier:

• sandbox → 0.1 (90% discount on all gas charges)
• all others → 1.0

10. Database Provisioning

Two modes:

Phase 2 (Vault, preferred for MongoDB):

• Vault feature-flagged via VAULT_ENABLED=true + isVaultHealthy()
• Creates Vault MongoDB dynamic credentials role per service
• Creates Vault policy + K8s auth binding + ServiceAccount
• Returns Vault Agent pod annotations for sidecar injection
• Pod reads credentials from /vault/secrets/ at runtime (short-lived, auto-rotated)
• Falls back to Phase 1 on any Vault error

Phase 1 (static K8s secrets, fallback):

• Calls provisionDatabases() per database type
• MongoDB: creates svc_{service}_{env} user with readWrite on {service}-{env} DB
• Redis: creates ACL user with svc_{service}_{env} username
• Neo4j: creates user with svc_{service}_{env} username
• Postgres (iec-builder#231): creates login role svc_{service}_{env} owning database {service}-{env} (or name:), CONNECT revoked from PUBLIC. Always Phase 1 — no Vault path yet. DB_POSTGRES_ADMIN_URI unset → [warn] in the build log and nothing injected (it never falls back to an unauthenticated URL)
• Creates K8s Secret {service}-db-{type} with connection string
• Requires: DB_MONGODB_ADMIN_URI, DB_REDIS_ADMIN_URL, DB_NEO4J_ADMIN_URI, or DB_POSTGRES_ADMIN_URI

Env vars injected:

Koko DB registration: Stores the connection string in Koko so koko-db CLI can discover it. Only registers when authenticated credentials exist.

Database sharing: If spec.databases[].sharedWith is declared, stores sharing config on the service record. Consumer services can then access the DB via spec.consumesDatabase.

11. Consumed Database Provisioning

When: catalog.consumesDatabase is non-empty (cross-service DB access).

Scope grant check:

What: Creates a svc_{consumer}_on_{owner}_{env} MongoDB user with the declared access level (readOnly or readWrite) on the owner's database.

12. Object Storage Provisioning (catalog 0.3.0+)

Requires: Vault enabled + healthy. MinIO running at MINIO_HOST:MINIO_PORT.

Steps:

1. Create bucket via MinIO SDK (idempotent)
2. Set quota via mc quota set CLI
3. ensureVaultRole — creates Vault MinIO role with an inline IAM policy scoped to the bucket
4. ensureVaultPolicy — adds MinIO creds path to the Vault policy
5. ensureKubeAuthRole — ensures K8s auth can issue tokens for the service's ServiceAccount
6. Returns Vault Agent annotations for sidecar injection

Tiers and gas:

Env vars (injected via Vault Agent to /vault/secrets/storage): S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_ENDPOINT, S3_REGION

13. Dependency Resolution (catalog 0.4.0+)

Two formats in catalog-info.yaml:

Legacy spec.internalDependencies (no scopes):

• No Bio-ID scope check
• Always resolves via Koko to K8s or Janus URL
• Injects {SERVICE}_URL + {SERVICE}_CHARGE_MODE

Unified spec.dependencies (with scopes):

1. If scopes non-empty: check Bio-ID scope grant status
2. denied/revoked → block (hard fail after OAuth provisioning pass), no URL injected
3. pending → block (awaiting approval), no URL injected
4. none → record as a potential blocker but still resolve + inject the URL, then let the grant-creation pass decide: target has no Bio-ID module → fail-open (proceed); same-org → auto-approved (proceed); cross-org with a module → creates a pending request that re-blocks. Injecting the URL here is what makes fail-open / auto-approved deps actually functional.
5. unreachable → warn + proceed (URL injected)
6. approved → proceed (URL injected)
7. transport: gateway → no URL injection (consumer calls Janus public gateway)
8. transport: direct → inject K8s internal URL (same-org only; cross-org silently routes via Janus)
9. transport: janus → inject Janus proxy URL ({JANUS_URL}/i/{service})

URL injection also requires the dependency to be registered in Koko — a dep that isn't in the registry resolves to no URL (warn), independent of grant status.

URL formats:

Charge mode injected alongside URL: {SERVICE}_CHARGE_MODE = callerOrg | service

13b. Provider Webhook Secrets

When: a spec.dependencies entry declares webhooks.path.

What: reads the target's webhook contract from Koko (metadata.webhookProvider), computes the consumer's receiver URL (verified custom domain for the environment, else the platform hostname, plus the path), mints the builder's Bio-ID client_credentials token (BIO_CLIENT_ID / BIO_CLIENT_SECRET in /opt/iec-builder/.env), and POSTs { orgSlug, url, consumerService, environment } to the provider's provisionEndpoint on its public platform hostname. The returned secret is merged into the provisioned env vars under the contract's secretEnv and reaches the pod through {service}-managed-secrets.

A value the developer set with tawa config set (plain or --secret) wins and is not overwritten; a mismatch is a [warn]. The secret is never written to the build log, Koko, or the deploy snapshot.

Fail behavior: Fail-open — every failure is a [warn] linking https://tawa.insureco.io/reference/provider-webhook-secrets.

Provider side: a provider's own deploy syncs spec.webhookProvider to Koko as metadata.webhookProvider (null when absent) during service registration (step 22).

14. OAuth Client Provisioning

When: catalog.authMode == 'sso' OR catalog.authMode == 'service-only' OR service has any scoped dependencies.

What: Calls Bio-ID POST /api/admin/oauth-clients (idempotent — upserts).

Redirect URI registered:

• Sandbox: https://{service}.sandbox.tawa.pro/api/auth/callback
• Prod: https://{service}.tawa.pro/api/auth/callback
• Custom domain (if dnsVerified): https://{customDomain}/api/auth/callback

What is merged vs overwritten on an existing client: the builder reads the client first, then sends the union of what it derives and what Bio-ID already holds for redirectUris, allowedScopes and allowedGrantTypes — so a deploy only ever adds to those three, and a scope an admin granted with tawa oauth update --scopes survives. Preserved entries are named in the build log (OAuth scopes preserved from Bio-ID (not derived from the catalog): …). If the client cannot be read, the scope and grant lists are omitted from the PATCH entirely rather than narrowed — fail-open, with a warning. Catalog-declared fields (spec.auth.passport → passportConfig, headlessEnabled, allowedEmbedOrigins) are written from the catalog every deploy. See auth and iec-builder#159.

Env vars injected:

• BIO_CLIENT_ID — OAuth client ID ({service}-{environment})
• BIO_CLIENT_SECRET — OAuth client secret (rotated on each deploy). A credential, so it rides the managed secret, not the Helm command — see Step 18
• BIO_ID_URL — Bio-ID public URL
• BIO_ID_CALLBACK_URL — Full callback URL for this service

14b. Claude Access

When: an OAuth client was provisioned in step 14, Anthropic federation is configured on the builder, and that client has Claude access (claudeAccessEnabled) switched on in Bio-ID.

What: the builder reads the client back from Bio-ID. If the flag is on it finds, or creates, the client's own Anthropic service account and federation rule — the same idempotent call the console switch makes, so a deploy also repairs either one if it has gone missing — and injects:

• ANTHROPIC_FEDERATION_RULE_ID, ANTHROPIC_ORGANIZATION_ID, ANTHROPIC_SERVICE_ACCOUNT_ID, ANTHROPIC_WORKSPACE_ID

These are identifiers, not credentials, so they travel as plain --set env.* values, not through the managed secret. All four are reserved against tawa config set.

Build log: Claude access provisioned for {clientId}: rule fdrl_…, service account svac_…. Injected …

Fail behavior: Fail-open. Bio-ID unreadable, Anthropic unreachable, or an existing rule that is not what the client should have: one [warn] line, the variables are not injected, the deploy continues. A client with the flag off, or a builder with no federation configured, logs nothing.

Full reference: Claude Access.

15. Scope Grant Creation

When: Service has scoped dependencies (hasScopedDeps = true).

For each dep with scopes that doesn't have an approved grant:

1. POST /api/admin/scope-grants in Bio-ID
2. Same-org → auto-approved immediately. "Same org" compares the target's Bio-ID module owner with the requesting service's org: its own module owner if it declares spec.modules, otherwise the org of its OAuth client ({service}-{env}, set from spec.owner on deploy). A consumer does not need modules of its own.
3. Cross-org → status pending — target org must approve in Console → Permissions → API Access
4. Bio-ID unreachable → fail-open (auto-add to approved set, proceed)
5. Target module not registered → fail-open

For each dep whose approved grant lacks a declared scope (approved_missing): the same POST /api/admin/scope-grants with the full declared scope list, which widens the existing grant for same-org — see Adding a scope to an existing dependency.

Never for a revoked grant — see Revoked grants.

Block enforcement: After all scope grant creation attempts, any deps still in the blocked list (pending/denied/revoked/none/approved_missing that weren't auto-approved, widened or fail-open) cause a hard fail.

Important: A pending grant does NOT inject the {SERVICE}_URL — the resolver continues past URL injection when a dep is blocked. So a stuck-pending grant looks like "the URL never got injected" at runtime, but the root cause is the unapproved grant. Approve it, then redeploy.

Surfacing (so you know exactly what to approve): Each blocking dep is emitted as its own build log line and persisted to a structured build.blockedGrants array on the build record:

[blocked] relay [relay:send] — grant pending (grantId: a7d21c07-…). Approve at https://tawa.insureco.io/console/permissions/api-access

// build.blockedGrants[]
{ "service": "relay", "scopes": ["relay:send"], "status": "pending",
  "grantId": "a7d21c07-…", "approveUrl": ".../console/permissions/api-access",
  "reason": "Scope grant pending for relay [relay:send] (grantId: …) — approve at …" }

When the grant was opened by this very deploy and Bio-ID did not auto-approve it, the reason says why, in Bio-ID's words:

Scope grant pending for x [x:read] (grantId: …) — not auto-approved as same-org:
cross-org: 'my-svc' is owned by 'acme', the target by 'insureco'. Fix: …

Other reasons: no org is recorded for '<svc>' (neither a module owner nor an org on its OAuth client) and the target module has no owner org recorded in Bio-ID. Either one on two services you know share an org is a platform bug, not a grant to approve by habit.

Correction (2026-09-30): until iec-bio#160, same-org was decided from the requesting service's module only. A consumer without spec.modules — portal, and most relay callers — therefore never matched, and every same-org scoped dependency blocked its first deploy on a pending grant someone had to approve by hand (iec-builder#204). Grants already approved by hand stay approved; nothing needs redoing.

The Deploy blocked: … error reply also carries the grantId inline. Approve via the console, or PATCH /api/admin/scope-grants/{grantId} with { "status": "approved" } (super_admin or owner of the target service).

Adding a scope to an existing dependency

Adding a scope to a dependency that already has an approved grant ([webmaster:read] → [webmaster:read, webmaster:dns]) does not open a second grant. Bio-ID keeps one grant per service pair + credential, and since iec-bio#161 POST /api/admin/scope-grants against an approved grant behaves like this:

The builder does this for you (iec-builder#221). When the resolver finds an approved grant that does not cover every declared scope, it records status approved_missing — never pending; nothing is pending and nobody has been asked — and the grant-creation pass POSTs the full scope list with the same requestingServiceId, targetServiceId, credentialType: "oauth_client" and credentialId ({service}-{env}):

On a block, build.blockedGrants[] carries the detail:

{ "service": "iec-webmaster", "scopes": ["webmaster:read", "webmaster:sync", "webmaster:dns"],
  "status": "approved_missing", "grantId": "…", "missingScopes": ["webmaster:dns"],
  "needsApprovalBy": "insureco",
  "reason": "Scope grant for iec-webmaster [...] is approved, missing webmaster:dns — needs approval by insureco (grantId: …). Bio-ID: Approved scope grant … does not include: webmaster:dns. … Fix: …",
  "fix": "Fix: an admin of 'insureco' revokes the approved grant (`tawa scopes revoke …`), you run `tawa scopes request iec-webmaster webmaster:read,webmaster:sync,webmaster:dns` …" }

needsApprovalBy is the target service's owning org from Koko; when Koko did not say (a gateway dependency), it reads the org that owns '<service>'.

So, same-org: just redeploy — the grant is widened in place, with no revoke and no gap in access. Do not revoke the grant to make room for a new one: that cuts access until the new grant exists, and the builder will not re-request a revoked grant.

Cross-org: the target org has to review the wider set. Bio-ID will not widen a cross-org grant, so the fix the block hands you is: an admin of the target org revokes the grant (tawa scopes revoke <grantId>), you request the full list (tawa scopes request <service> <all,scopes>), and they approve the new grant (tawa scopes grant <grantId>); then redeploy. Access to the dependency is off in between.

Correction (2026-10-02): before iec-builder#221 the builder reported an approved grant missing a scope as grant pending (grantId: …) and never asked Bio-ID to widen it, so every deploy blocked and the documented workaround was a manual POST /api/admin/scope-grants. That is no longer needed.

Revoked grants

When a dependency's only grant is revoked, the deploy blocks and the builder does not re-request it. Bio-ID treats only a pending or approved grant as existing, so a re-request creates a fresh grant — and for a same-org pair approves it at once. The builder would silently undo a revocation someone made on purpose. The block says exactly that:

Access to relay [relay:send] has been revoked (grantId: …). The builder does not re-request a
revoked grant: a same-org request is approved at once, which would silently undo the revocation.
Fix: ask the owner of 'relay' why it was revoked. If access should be restored, run
`tawa scopes request relay relay:send` (same-org is approved at once; cross-org waits for an
admin of 'relay' to run `tawa scopes grant <grantId>`), then redeploy. Otherwise remove 'relay'
from spec.dependencies. See https://tawa.insureco.io/reference/build-pipeline

Once a new grant is approved, the gate finds it ahead of the revoked one and proceeds.

16. Module Registration (Bio-ID)

When: catalog.modules is non-empty.

What: Registers each module with Bio-ID. Returns a per-service API key injected as BIO_INTERNAL_KEY into the pod (scoped to this service, not the builder's master key).

Fail behavior: Non-blocking — logs warning if registration fails.

17. userAccess Provisioning (catalog 0.5.0+)

When: catalog.userAccess is declared.

What:

• Injects BIO_USERS_URL = https://bio.tawa.pro/api/v2/users
• Provisions a sak_ service API key in Bio-ID and injects it as BIO_SERVICE_KEY
• Registers the user access scope in Koko (for tawa-web Console display)
• scope: cross-org → lists the service as an allowed consumer in the Koko record

The key is keyed {service}-{environment} and provisioning is idempotent — Bio-ID returns the same key on every subsequent deploy. It is persisted (encrypted) to the service record and flows through the managed-secrets K8s secret path, so it survives redeploys rather than existing only as a Helm --set env.* argument.

Two conditions gate it, and a failure of either is non-fatal — the build continues and the warning is the only signal:

So a service can deploy green, hold BIO_USERS_URL, and still have no key — check the build log for the User access service key provisioned line rather than assuming.

Correction (2026-09-14): this section previously said BIO_SERVICE_KEY is NOT injected and told you to provision one separately. That was wrong, and wrong in the costly direction: a reader concluded the platform had no service-level path to the user directory and went looking for a workaround. The builder has provisioned this key since spec.userAccess shipped (builder.ts, the catalog.userAccess block). user-access.md always documented it correctly — that page is the one to trust on this.

A client-credentials token is not a substitute. A service's own OAuth token (BIO_CLIENT_ID/BIO_CLIENT_SECRET, grant_type=client_credentials) carries client_id, orgId, orgSlug and scope — but no sub, no roles, and no userType. Bio-ID's directory routes gate on caller identity, so verified against prod on 2026-09-14 it is refused everywhere:

spec.userAccess + BIO_SERVICE_KEY is the supported way for a service to read the user directory with no user in the request. See user-access.md.

18. Helm Deploy

Command pattern:

helm upgrade --install {service} {chartPath} \
  --namespace {namespace} \
  --create-namespace \
  --set image.tag={imageTag} \
  --set image.repository={registry}/{service} \
  --set service.port={port} \
  --set health.endpoint={healthEndpoint} \
  --set secretRef={service}-managed-secrets \
  --set env.KEY=VALUE  ... (service.config + the NON-credential provisioned vars)
  --set resources.requests.cpu={req} --set resources.requests.memory={req} \
  --set resources.limits.cpu={tier} --set resources.limits.memory={tier} \
  --set strategy.type=RollingUpdate \
  --set-string strategy.rollingUpdate.maxSurge=100% \
  --set strategy.rollingUpdate.maxUnavailable=0 \
  --wait --timeout ${HELM_TIMEOUT:-15m} --history-max 5
  # + Vault Agent annotations if Vault active
  # + NetworkPolicy labels for direct-transport deps

Resource requests and limits. Limits always come from insureco.io/pod-tier, and so does hosting gas. Requests default to the tier's values (see the table in Gas & Wallet), and a service may move them with insureco.io/memory-request and insureco.io/cpu-request (iec-builder#193). Formats, floors, the tier-limit ceiling and the warn-and-fall-back rule are in catalog-info → memory-request / cpu-request. Every deploy writes one line naming the source of each request:

Resource requests: cpu 400m (xlarge tier default), memory 512Mi (insureco.io/memory-request); limits: cpu 4, memory 4Gi (xlarge tier)

These apply to the service container only. When Vault is active, the Vault Agent sidecar gets its own requests through its injector annotations (iec-builder#192), logged as Vault Agent sidecar requests: cpu 10m, memory 48Mi (limits unchanged). A custom chart receives the same --set resources.*; if an override is declared and its templates never read .Values.resources, the build warns that the requests are passed but not applied.

Every replica rolls at once, with zero downtime (iec-builder#171). Before this change the strategy was maxSurge: 1, maxUnavailable: 0, so a 2-replica service replaced its pods one after the other. On bio, which pulls a 216 MB image at about 1 MB/s, that was two 3–4 minute pulls back to back: about 7 minutes of rollout. With maxSurge: 100% a whole new ReplicaSet starts beside the old one, every new pod pulls and starts in parallel, and old pods are removed only as new ones become Ready (maxUnavailable: 0). A rollout now costs one pod's startup, not one per replica.

The cost is transient. For the length of the rollout the service reserves up to twice its replicas' resource requests. If no node has room, the surge pods sit Pending until the autoscaler adds capacity: slower, never unsafe. The same default is in helm/default-service/values.yaml, and the --set flags also apply to custom charts that read .Values.strategy. --timeout stays at 15 minutes as headroom for that Pending case and for a cold node pulling a large image.

One Helm run per release at a time. The Helm step (status check, failed-release cleanup, upgrade) runs under a per-{namespace}/{service} lock in Redis. A second build for the same release — a different commit, since same-commit races are caught at trigger time — waits, logging [helm] another deploy of X is in progress — build <id> — waiting… once a minute, then runs in order. It does not fail with release: already exists. Waiting is bounded at 30 minutes; past that the waiter fails naming the holder, because a lock held that long is a stuck deploy, not a slow one. A crashed holder's lock expires on its own (iec-builder#110).

Helm chart priority:

1. .iec.yaml helmChart field
2. service.helmChart stored in builder DB (set via tawa services update --helm-chart)
3. Auto-discovered: helm/Chart.yaml, helm/{service}/Chart.yaml, helm/*/Chart.yaml, chart/Chart.yaml
4. Built-in helm/default-service chart (fallback)

Secrets: the K8s Secret {service}-managed-secrets, mounted via envFrom.secretRef, is rebuilt from scratch on every deploy from two inputs:

1. the developer's secrets — tawa config set --secret, stored encrypted in service.secrets;
2. this deploy's provisioned credentials — the OAuth client secret, BIO_INTERNAL_KEY, and any provisioned connection string carrying a password (scheme://user:pass@host). Classified by key shape (SECRET, PASSWORD, TOKEN, CREDENTIAL, _KEY) or value shape; BIO_CLIENT_ID and credential-free URLs stay on the Helm command as --set env.*.

Provisioned credentials never enter service.secrets. The builder does not create entries in the developer's store; tawa config list shows only what the developer set. The Secret is applied from a 0600 temp-file manifest (nothing sensitive in kubectl's argv) and annotated tawa.pro/provisioned-keys naming the keys the builder put there. The pod checksum covers the merged data, so a rotated provisioned credential still rolls the pods.

What the build log says, every deploy:

Correction (2026-09-10): this page previously said every provisioned var, credentials included, went to Helm as --set env.KEY=VALUE. That was true until iec-builder#118 and put the values in the builder host's process table for the length of each deploy. Do not reintroduce that path. An earlier fix (#114) was reverted for writing provisioned values into the developer's service.secrets; the current design is what replaced it. BIO_SERVICE_KEY is the one deliberate exception that still persists itself into service.secrets.

NetworkPolicy: Pods default to receiving traffic only from tawa.pro/role: janus namespace (Janus sidecar). Direct-transport dependencies label the namespace so NetworkPolicy allows direct pod-to-pod traffic.

Egress NetworkPolicy (spec.egress.denyPrivate): egress is unrestricted unless the catalog declares spec.egress.denyPrivate: true. Then, just before this step, the builder computes the allowed destinations (src/services/egress-policy.ts) and passes them as --set-json 'networkPolicy.egress={"enabled":true,"rules":[...]}'; the default chart adds Egress to policyTypes and renders the rules. The pod may reach the internet minus 10/8, 172.16/12, 192.168/16, 169.254/16 and 100.64/10, plus cluster DNS, its own namespace, the platform services, its declared dependencies, ingress-nginx, Vault (when it runs an agent), the builder's backing-store hosts for what it declares (DB_MONGODB_HOST, DB_REDIS_HOST, DB_NEO4J_HOST, MINIO_INTERNAL_HOST — which must be IP addresses for a rule to be written), the egress-group proxy, and spec.egress.allow. The build log lists each destination; a custom chart gets a [warn] because only the default chart renders the value. Undeclared, nothing is passed and the chart renders byte-identically — enforced by golden fixtures in iec-builder. Full reference: catalog-info → spec.egress.denyPrivate.

Deploy snapshot: immediately after helm upgrade succeeds, the builder POSTs one deploy manifest to Koko (POST /api/services/:id/manifests). This is the only manifest write in the pipeline, so a stored manifest always describes a build that shipped. It carries:

The call is fire-and-forget — it never blocks or fails a deploy — but it is not silent: if Koko rejects the manifest, the build log gets [warn] Deploy snapshot was not stored in Koko: <Koko's error>. Treat that line as a bug report against the builder↔Koko contract; deploy history and the SDK upgrade gate both depend on this write.

19. DNS Configuration

When: ENABLE_DNS_MANAGEMENT=true.

What: Creates or updates a Cloudflare CNAME record:

• {service}.{env}.tawa.pro (sandbox/uat) or {service}.tawa.pro (prod) → INGRESS_TARGET

Fail behavior: DNS errors are non-fatal — logged as [warn], deploy continues.

After DNS: registers domain binding in Koko.

20. Post-Deploy Pod Health Check

What: kubectl get pods -n {namespace} + recent events. Logged to build output. Stored as build.diagnostics if issues detected.

Health reports the running commit. Alongside the pod check, the builder makes one HTTPS GET, with a 10s timeout, to the service's public platform hostname (https://{service}.tawa.pro{healthEndpoint}, or .sandbox. / .uat.) after the rollout, and parses the JSON. If the body has no commit equal to the deployed SHA (the GIT_COMMIT it injected), the build log gets one line:

[warn] /health does not report the running commit. Return { status, commit: process.env.GIT_COMMIT } — see https://tawa.insureco.io/reference/catalog-info

The result is stored as build.healthReportsCommit (true, false, or null when it could not be checked). worker and internal: true services are skipped, recording null with no warning. An unreachable endpoint, a timeout or a non-2xx answer also records null with no warning. A 2xx body that is not JSON warns once. Handlers: catalog-info → health-endpoint.

Fail behavior: Non-fatal — diagnostics only. Does not block subsequent steps, and the commit check never fails a deploy.

20b. Vault Secret Readback

When: VAULT_ENABLED=true. Runs after the pod health check, before deploy-gated tests.

What: Confirms the Vault Agent actually rendered the declared secrets into the pod — not just that the vault-init container came up. For each declared database + object storage, the builder kubectl execs into a Running pod and cats the rendered file, verifying the required keys are present with non-empty values:

Result (healthy | degraded | missing | skipped | error) is stored as build.vaultVerification and cached on service.vaultVerification, and surfaced via GET /services/:name/vault-status (powers the console vault health badge).

Fail behavior:

• Re-rolls only (auto-rotation and manual rotation/refresh deploys): a missing or degraded readback hard-fails the build, so the credential rotation clock does NOT advance on empty secrets. This is the post-reroll safety gate.
• Normal deploys: non-fatal — result is recorded and badged but never blocks the build.
• error (no ready pod / kubectl unavailable) and skipped (nothing declared) never block.

21. Deploy-Gated Tests (catalog 0.5.0+)

When: IEC_TEST_URL is set AND (catalog.tests is non-empty OR catalog.walkthrough is declared).

Target URL: Internal cluster URL (http://{service}.{namespace}.svc.cluster.local:{port}) — avoids external DNS propagation latency.

Flow:

1. POST {IEC_TEST_URL}/runs with test specs
2. Poll GET /runs/{id} every TEST_RUNNER_POLL_INTERVAL_MS (default 5s)
3. Timeout after TEST_RUNNER_MAX_POLL_MS (default 10 minutes)

Suites (each requested only when it has specs):

The two browser suites share one posture, and the prod exemption is narrow for both: it applies only when they are the only failures. A run where smoke also failed fails the build as usual.

e2e used to skip prod, which meant it never ran at all — nothing on this platform deploys to sandbox or UAT. Fixed in iec-builder#161, together with the shape: the builder parsed an entry as { name, spec: <string> } while iec-test required { name, steps: [...] }, so a catalog that declared e2e got Failed to create test run: 400. Steps are now validated at catalog-parse time, before the image is built. See Tawa Test for the contract.

A walkthrough run also sends the resolved commitSha, and on completion the build log gets Walkthrough: <url> pointing at the published bundle page. An e2e run logs its flow count and which posture applies before the build starts.

Fail behavior: Hard fail if tests fail. Deployment NOT rolled back.

22. Koko Service Registration

What: registerOrUpdateServiceInKoko — registers/updates the service in the service registry with:

• Routes (path, methods, auth, gas cost)
• Owner (org slug)
• Upstream — http://{service}.{namespace}.svc.cluster.local:{port}, where port is insureco.io/port from catalog-info.yaml (its framework default when omitted — static is 8080), falling back to the cached service record, then 3000. This is the same port Helm gives the Service (Step 18), and it is what Janus /i/ dials — Janus proxies internal calls to the Koko upstream, not to a URL of its own. The builder writes the resolved catalog port back to the service record after the deploy so the fallback stays current.
• Health endpoint — insureco.io/health-endpoint from catalog-info.yaml (falling back to the cached service record, then /health). This is the path iec-pulse probes, so a stale value here shows as 0% uptime on the status board.
• Pod tier + storage tiers
• Direct-transport dependency labels (for NetworkPolicy)

23. Queue + Schedule Bindings

What: registerQueueBindings / registerScheduleBindings — tells Koko the upstream K8s URL for iec-queue and iec-cron to call when delivering jobs or firing schedules. It is the same URL, and the same declared port, as the Step 22 registration.

Correction (2026-09-27): before iec-builder#187 / #188, Steps 22 and 23 registered the upstream on service.port || 3000 and ignored insureco.io/port, while Helm (Step 18) used the declared port. A service on any other port — refund-travel-rater on 3215, binddesk-api on 4000, static sites on 8080 — was registered at :3000, so Janus /i/ and iec-cron callbacks to it hung. Such services must be redeployed once after the builder fix; Koko is not backfilled.

Then it reconciles. After registering, the builder lists Koko's bindings for {service}-{environment} and deletes any whose name the catalog no longer declares. Remove a schedule from catalog-info.yaml and the next deploy removes it from Koko too — it stops firing.

The prune runs even when the catalog declares no queues and no schedules, because a catalog that dropped its last schedule is exactly the case that used to orphan one. It is scoped strictly to this service's own namespace: Koko's list routes have no server-side namespace filter, so the list is filtered on each record's namespace field, never by splitting the composite {namespace}-{name} id. Another service's bindings are never touched.

DELETE /services/:serviceId does the same with an empty keep-list, across prod, sandbox, uat and dev, before deleting the builder's own record.

Why this exists (iec-builder#165). The builder only ever added bindings. isc-split-pay was renamed to iec-split-pay; seventeen days later iec-cron was still firing seven schedules at isc-split-pay-prod, each one debiting 5 gas before the callback and getting a 404 back — about 150 gas a day for nothing. Rename still has no builder event: it is a new service plus an old one nobody deleted. Rename = deploy the new name, then tawa services delete the old one.

Fail behavior: Fail-open. Partial registration is logged. If Koko cannot be listed, nothing is deleted and the build log gets a [warn] naming what may still be registered. Neither step ever blocks a deploy.

24. Uptime Monitor (iec-pulse)

When: IEC_PULSE_URL is set.

What: Registers a monitor for https://{service}.{env}.tawa.pro{healthEndpoint}.

• prod: 30s poll interval
• sandbox/uat: 60s poll interval

Fail behavior: Fire-and-forget — never blocks.

spec.seo: what the builder keeps and who can read it (iec-builder#194, #206)

Every deploy that parses a catalog writes spec.seo to the service record as seoSpec (enabled, entity, topic, pages), or removes seoSpec if the catalog no longer declares it. seoSpec is separate from catalogSpec because the builder only writes catalogSpec for services that declare databases, and most frontends declare none.

GET /seo/services?env=prod (env defaults to prod) lists every service with seoSpec.enabled:

{ "success": true, "data": [
  { "service": "raterspot-web", "env": "prod", "seo": { "enabled": true, "entity": "raterspot" },
    "domains": ["raterspot.com"],                        // custom domains dnsVerified for this env
    "lastDeploy": { "commit": "…", "at": "…" } } ] }     // the latest completed build for this env

iec-webmaster's hourly service-sync reads this route to catch a deploy whose POST /hooks/deploy it missed.

This is the first builder route that accepts a platform service's own Bio-ID login:

The route is read-only and returns no credentials. The middleware is authenticateServiceOrUser() (src/middleware/service-token.ts). Use it only on routes whose answer is safe for any insureco service.

Fail-Open vs Fail-Hard Quick Reference

Environment Variables

All builder env vars with their defaults and purposes:

Auto-Injected Platform Env Vars (Always Available in Pods)

These are injected by the builder for every deploy — services do NOT need to declare them in internalDependencies:

What Gets Written Where

Common Failure Patterns

"[warn] /health does not report the running commit"

The deploy succeeded; your health response has no commit equal to the deployed SHA (build.healthReportsCommit: false). Return commit: process.env.GIT_COMMIT ?? null from the health handler and redeploy. Check that a Next.js handler is force-dynamic (otherwise the value is read at build, when it is unset) and that the declared insureco.io/health-endpoint is the path you edited. Handlers: catalog-info.

"Deploy gate: insufficient gas reserve"

Org wallet balance is below the 3-month reserve. Run tawa wallet buy N to top up, then redeploy.

"Deploy gate: catalog spec.owner X does not match registered service owner Y"

Someone changed spec.owner in catalog-info.yaml. Either revert it, or use tawa services update <id> --org <new-org> if the transfer is intentional.

"Deploy blocked: Access to X has been denied/revoked"

The scope grant for a spec.dependencies entry was explicitly denied or revoked in Bio-ID. Check Console → Permissions → API Access. The builder never re-requests a revoked grant — see Revoked grants for why and what to run.

"grant approved, missing X — needs approval by <org>"

A dependency gained a scope its approved grant does not cover, and the grant is cross-org, so Bio-ID refused to widen it (409). Same-org grants are widened automatically and never show this. Follow the Fix: in the block — revoke, request the full list, approve — see Adding a scope to an existing dependency.

"Deploy blocked: Scope grant pending for X [scopes] (grantId: …)"

A cross-org spec.dependencies entry created a grant request that nobody approved yet. The build log line [blocked] X [scopes] — grant pending (grantId: …) and build.blockedGrants[] name the exact grant. Approve it in Console → Permissions → API Access, or PATCH /api/admin/scope-grants/{grantId} { "status": "approved" } (super_admin / target-service owner), then redeploy. Note: while pending, {X}_URL is not injected — fix the grant, don't hardcode the URL.

"Convention check failed: N blocking violation(s)"

Source code violates a Koko gate rule (e.g., deprecated SDK usage). Fix the violations or use tawa deploy --skip-convention-check --reason "...".

"Config preflight failed: missing required config vars: KEY1, KEY2"

catalog.configDeclarations lists required vars that aren't set. Run tawa config set KEY1=... KEY2=.... On a first deploy this needs the service record to exist first — see the next entry.

"Service 'X' not found" from tawa config set / tawa config list

Not a typo and not a permissions problem: the service has never been registered. The record is created by tawa link, or automatically by the first tawa deploy — config routes have nothing to attach values to until then. Fix: tawa link (or deploy once), tawa config set …, then deploy again to inject the values. Full ordering: First Deploy: the deploy → config → deploy ordering.

If the service is already deployed, check the name matches metadata.name in catalog-info.yaml and that you are in the owning (or an authorized) org.

"Undocumented upgrade: X Y → Z has no changelog entry"

An SDK was bumped without a Koko changelog entry. Run tawa changelog --sdk X to document it.

"Deploy-gated tests failed"

Post-deploy smoke tests failed. The pod is still running — use kubectl logs to debug. Re-deploy after fixing will re-run tests.

Pod stuck in CrashLoopBackOff after deploy

Check build.diagnostics in the builder DB or run kubectl describe pod in the namespace. Common causes: missing env vars, incorrect health endpoint, DB connection failure on startup.

Related: Builder Logs

The builder's own PM2 logs are redacted (write-time + read-time), rotated, and exposed through a platform-admin API. See Builder Logs — Redaction, Rotation & Admin API for the GET /platform/logs and GET /platform/logs/tail contracts.