Build & Deploy Pipeline

The iec-builder executes a fixed, ordered pipeline for every tawa deploy. Understanding each step — what it does, when it fails hard vs. fails open, and what it writes to each system — is essential for debugging deploys and hardening the platform.

Pipeline Overview

tawa deploy --prod
     │
     ▼
 iec-builder (builder.ts: executeBuild)
     │
     ├─ PRE-BUILD
     │   ├─ Clone/extract
     │   ├─ Catalog parse
     │   ├─ Dockerfile strategy
     │   ├─ Deploy gate (wallet reserve)
     │   ├─ Config preflight
     │   └─ Convention check / SDK upgrade gate
     │
     ├─ BUILD
     │   ├─ docker build
     │   └─ docker push → registry
     │
     └─ DEPLOY  (ENABLE_K8S_DEPLOY=true)
         ├─ Namespace + Koko namespace registration
         ├─ Database provisioning (Vault or static K8s secrets)
         ├─ Consumed database provisioning
         ├─ Object storage provisioning (Vault, catalog 0.3.0+)
         ├─ Dependency resolution (Koko + Bio-ID scope grants)
         ├─ OAuth client provisioning
         ├─ Scope grant creation
         ├─ Module registration (Bio-ID)
         ├─ userAccess provisioning
         ├─ helm upgrade --install
         ├─ Deploy snapshot (Koko, fire-and-forget)
         ├─ DNS configuration (Cloudflare)
         ├─ Post-deploy pod health check
         ├─ Deploy-gated tests (iec-test, catalog 0.5.0+)
         ├─ Service registration in Koko
         ├─ Queue + schedule binding registration
         ├─ Role + registration metadata (catalog 0.6.0+)
         └─ Uptime monitor (iec-pulse, fire-and-forget)

Status Flow

queued → cloning → building → pushing → deploying → testing → completed
                                                            ↘ failed (any step)

Step-by-Step Reference

1. Clone / Extract

What: Clones the repo at the specified commitSha using the branch as a hint. If a tarball was uploaded (via POST /builds/upload), it extracts that instead.

Auth: Forgejo repos get a PAT injected (forgejo-token:TOKEN@host). GitHub SSH URLs are converted to HTTPS with a GitHub PAT. Both are handled by injectGitCredentials().

Output: Actual commit SHA is resolved and backfilled to the build record. Workspace at {WORKSPACE_DIR}/{buildId}.

Fail behavior: Hard fail — no source = no build.

2. Read .iec.yaml (monorepo config)

What: Reads .iec.yaml from the repo root (or {service.appPath}/.iec.yaml). Controls:

• appPath — subdirectory containing the service
• dockerfile — custom Dockerfile path
• buildContext — custom build context (relative to appPath)
• helmChart — custom Helm chart path

Priority for appPath: service.appPath (DB) > .iec.yaml > inferred from service.dockerfilePath.

Fail behavior: Missing .iec.yaml is fine — defaults apply.

3. Parse catalog-info.yaml

What: Reads catalog-info.yaml from {workDir}/{effectiveAppPath}. Parses all fields: framework, databases, routes, dependencies, auth, storage, schedules, queues, etc.

Catalog version:

• Current: 0.6.0 — supports modules, registration metadata, role IDs
• Minimum: 0.2.0 — old catalogs below this are rejected (when ENFORCE_MINIMUM_CATALOG_VERSION=true)
• 0.3.0 — storage buckets
• 0.4.0 — unified spec.dependencies with Bio-ID scope grants
• 0.5.0 — deploy-gated tests, spec.userAccess, config declarations
• 0.6.0 — spec.userAccess.roles, spec.registration

Framework auto-detection: If no catalog, the builder sniffs package.json for Next.js, Express, Hono, Fastify, etc. and constructs a minimal catalog.

Fail behavior: Missing catalog is a warn (not a hard fail) — deploy proceeds with defaults.

4. Dockerfile Strategy

Decision tree:

Has explicit Dockerfile?
  (iec.yaml OR service.dockerfilePath != 'Dockerfile')
  OR catalog.customDockerfile == 'true'?
    YES → useCustomDockerfile = true
           ├─ patchCustomDockerfile:
           │   ├─ Replace non-numeric USER → USER 1001 (K8s runAsNonRoot)
           │   └─ Inject Vault entrypoint wrapper (when VAULT_ENABLED=true)
           └─ File not found → fall back to auto-generate
    NO  → ENABLE_AUTO_DOCKERFILE=true?
           YES → generateDockerfile(buildContext, catalog)
           NO  → use project Dockerfile as-is

Important: Services using a custom Dockerfile must add the Vault entrypoint wrapper manually unless the builder patches it. See patchCustomDockerfile().

Non-root validation: After Dockerfile resolution, validateNonRootDockerfile() checks that the final USER directive is a number ≠ 0. Fails hard. Exception: static framework (nginx runs as root internally).

Package manager: The generated Dockerfile auto-detects the lockfile and installs with the matching tool — yarn.lock → yarn, package-lock.json → npm, pnpm-lock.yaml → pnpm. For pnpm, the builder runs corepack prepare pnpm@<pinned> --activate (NOT a bare corepack enable pnpm), because bare enable pulls the latest pnpm (11.x), which requires Node ≥ 22.13 (node:sqlite) and crashes on the Node 20 build image. A repo's own packageManager field in package.json still overrides the pinned default — pin one there if your service needs a specific pnpm version. See dockerfile-generator.ts (PINNED_PNPM).

4b. Build-time env injection (.env.production)

When: Immediately before docker build, if the catalog declares build-time vars via insureco.io/env-vars.

The builder materializes the declared vars from the config store (tawa config set values + managed secrets) into a .env.production file at the build-context root. npm run build (CRA, Vite, and Next.js all read .env* files at build time) then bakes them into the client bundle.

Why this step exists: tawa config set values live in the builder DB, not the repo (whose .env is normally gitignored), and docker build runs without --build-arg. Without this step, REACT_APP_* / NEXT_PUBLIC_* vars compile to their hardcoded fallback (often localhost) and the deployed static bundle calls the wrong API URL. (Symptom: a static/CRA app whose login/API calls hit the app's own host or localhost instead of the real API.)

What is written: ONLY the keys listed in insureco.io/env-vars that have a value in config. These are public-by-design (they end up in the client bundle). The full secret set is never written here — non-declared secrets stay runtime-only (injected as K8s secrets at deploy). Value resolution: service.config merged over decrypted service.secrets.

Build log line: Wrote N build-time env var(s) to .env.production: KEY1, KEY2 (names only — values are never logged).

Belt-and-suspenders: client apps can also resolve their API base at runtime from window.location.hostname so they don't depend on build-time injection at all. Recommended for static SPAs as a fallback. See writeBuildEnvFile() in builder.ts.

5. Deploy Gate

When: ENABLE_K8S_DEPLOY=true AND WALLET_URL is set AND catalog was parsed.

Skipped for platform services: iec-wallet, iec-janus, tawa-web, koko, bio-id, iec-test (or any service in DEPLOY_GATE_SKIP_SERVICES env var).

Three checks (in order):

Fail-open: If the wallet service is unreachable, the gas reserve check returns "sufficient" and the deploy proceeds.

Reserve formula:

required = (hosting_gas_per_hour × 24 × 30 × 3) + (storage_gas_per_month × 3)
shortfall = max(0, required - current_balance)

Authorized orgs (multi-org collaboration): The deploy gate keys off catalog.owner (= spec.owner = the primary org). A service can additionally grant one or more authorized orgs whose devs get full-parity access — deploy, config, secrets, DB — without being moved into the owning org. The deploy gate is unaffected:

• The authorized-org dev never changes spec.owner, so the owner tamper guard still passes.
• Gas is still billed to the primary org's wallet (deriveWalletId(catalog.owner)).
• The authorized org does not need to be in approved_orgs — the allowlist checks catalog.owner, not the deploying user.

Authorization is a per-service access check only (user.org === service.org OR user.org ∈ service.authorizedOrgs), enforced on the build trigger, build views, and all service management routes. Manage it as an admin of the owning org:

GET    /services/:name/authorized-orgs                 # list owner + authorized orgs
POST   /services/:name/authorized-orgs { orgSlug }     # add   (owning-org admin only)
DELETE /services/:name/authorized-orgs/:orgSlug        # remove (owning-org admin only)

Only owning-org admins (or super_admin / platform service) may add or remove authorized orgs — an authorized org cannot add further orgs. A build triggered by an authorized-org dev is stamped with that dev's org, but build history is resolved through the service so the owning org and every authorized org share one view.

6. Config Preflight

When: catalog.configDeclarations is non-empty (catalog 0.5.0+).

What: Each declared config var with required: true must exist in service.config or service.secrets. Missing vars → hard fail with a list of what's missing and the tawa config set command to fix it.

Defaults: Vars with default: are injected into the deploy even if not set by the user (lowest priority — user config overrides).

7. Convention Check + SDK Upgrade Gate

Convention check: checkConventions() runs Koko gate rules against the source files.

• warn violations: logged but non-blocking
• block violations: hard fail (bypass with tawa deploy --skip-convention-check --reason "...")
• Skip is always logged to Septor audit regardless

SDK upgrade gate: For each detected package version bump vs. the previous deploy manifest:

• Checks Koko for a changelog entry at {package}@{version}
• No changelog entry + Koko reachable → hard fail
• Koko unreachable → warn and proceed (fail-open)
• Bypassed if --skip-convention-check was set

8. Docker Build + Push

Build:

docker build --platform linux/amd64 -t {DOCKER_REGISTRY}/{service}:{sha7} -f {dockerfile} {buildContext}

Context size check: Warns if build context exceeds threshold (avoids accidental large builds).

Push: docker push {imageTag} to the DigitalOcean registry.

Fail behavior: Hard fail on any non-zero exit.

9. Namespace + Koko Namespace Registration

Namespace: kubectl create namespace {service}-{environment} (idempotent).

Koko registration: Registers namespace with a gas multiplier:

• sandbox → 0.1 (90% discount on all gas charges)
• all others → 1.0

10. Database Provisioning

Two modes:

Phase 2 (Vault, preferred for MongoDB):

• Vault feature-flagged via VAULT_ENABLED=true + isVaultHealthy()
• Creates Vault MongoDB dynamic credentials role per service
• Creates Vault policy + K8s auth binding + ServiceAccount
• Returns Vault Agent pod annotations for sidecar injection
• Pod reads credentials from /vault/secrets/ at runtime (short-lived, auto-rotated)
• Falls back to Phase 1 on any Vault error

Phase 1 (static K8s secrets, fallback):

• Calls provisionDatabases() per database type
• MongoDB: creates svc_{service}_{env} user with readWrite on {service}-{env} DB
• Redis: creates ACL user with svc_{service}_{env} username
• Neo4j: creates user with svc_{service}_{env} username
• Creates K8s Secret {service}-db-{type} with connection string
• Requires: DB_MONGODB_ADMIN_URI, DB_REDIS_ADMIN_URL, or DB_NEO4J_ADMIN_URI

Env vars injected:

Koko DB registration: Stores the connection string in Koko so koko-db CLI can discover it. Only registers when authenticated credentials exist.

Database sharing: If spec.databases[].sharedWith is declared, stores sharing config on the service record. Consumer services can then access the DB via spec.consumesDatabase.

11. Consumed Database Provisioning

When: catalog.consumesDatabase is non-empty (cross-service DB access).

Scope grant check:

What: Creates a svc_{consumer}_on_{owner}_{env} MongoDB user with the declared access level (readOnly or readWrite) on the owner's database.

12. Object Storage Provisioning (catalog 0.3.0+)

Requires: Vault enabled + healthy. MinIO running at MINIO_HOST:MINIO_PORT.

Steps:

1. Create bucket via MinIO SDK (idempotent)
2. Set quota via mc quota set CLI
3. ensureVaultRole — creates Vault MinIO role with an inline IAM policy scoped to the bucket
4. ensureVaultPolicy — adds MinIO creds path to the Vault policy
5. ensureKubeAuthRole — ensures K8s auth can issue tokens for the service's ServiceAccount
6. Returns Vault Agent annotations for sidecar injection

Tiers and gas:

Env vars (injected via Vault Agent to /vault/secrets/storage): S3_BUCKET, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_ENDPOINT, S3_REGION

13. Dependency Resolution (catalog 0.4.0+)

Two formats in catalog-info.yaml:

Legacy spec.internalDependencies (no scopes):

• No Bio-ID scope check
• Always resolves via Koko to K8s or Janus URL
• Injects {SERVICE}_URL + {SERVICE}_CHARGE_MODE

Unified spec.dependencies (with scopes):

1. If scopes non-empty: check Bio-ID scope grant status
2. denied/revoked → block (hard fail after OAuth provisioning pass), no URL injected
3. pending → block (awaiting approval), no URL injected
4. none → record as a potential blocker but still resolve + inject the URL, then let the grant-creation pass decide: target has no Bio-ID module → fail-open (proceed); same-org → auto-approved (proceed); cross-org with a module → creates a pending request that re-blocks. Injecting the URL here is what makes fail-open / auto-approved deps actually functional.
5. unreachable → warn + proceed (URL injected)
6. approved → proceed (URL injected)
7. transport: gateway → no URL injection (consumer calls Janus public gateway)
8. transport: direct → inject K8s internal URL (same-org only; cross-org silently routes via Janus)
9. transport: janus → inject Janus proxy URL ({JANUS_URL}/i/{service})

URL injection also requires the dependency to be registered in Koko — a dep that isn't in the registry resolves to no URL (warn), independent of grant status.

URL formats:

Charge mode injected alongside URL: {SERVICE}_CHARGE_MODE = callerOrg | service

14. OAuth Client Provisioning

When: catalog.authMode == 'sso' OR catalog.authMode == 'service-only' OR service has any scoped dependencies.

What: Calls Bio-ID POST /api/admin/oauth-clients (idempotent — upserts).

Redirect URI registered:

• Sandbox: https://{service}.sandbox.tawa.pro/api/auth/callback
• Prod: https://{service}.tawa.pro/api/auth/callback
• Custom domain (if dnsVerified): https://{customDomain}/api/auth/callback

Env vars injected:

• BIO_CLIENT_ID — OAuth client ID ({service}-{environment})
• BIO_CLIENT_SECRET — OAuth client secret (rotated on each deploy)
• BIO_ID_URL — Bio-ID public URL
• BIO_ID_CALLBACK_URL — Full callback URL for this service

15. Scope Grant Creation

When: Service has scoped dependencies (hasScopedDeps = true).

For each dep with scopes that doesn't have an approved grant:

1. POST /api/admin/scope-grants in Bio-ID
2. Same-org → auto-approved immediately
3. Cross-org → status pending — target org must approve in Console → Permissions → API Access
4. Bio-ID unreachable → fail-open (auto-add to approved set, proceed)
5. Target module not registered → fail-open

Block enforcement: After all scope grant creation attempts, any deps still in the blocked list (pending/denied/revoked/none that weren't auto-approved or fail-open) cause a hard fail.

Important: A pending grant does NOT inject the {SERVICE}_URL — the resolver continues past URL injection when a dep is blocked. So a stuck-pending grant looks like "the URL never got injected" at runtime, but the root cause is the unapproved grant. Approve it, then redeploy.

Surfacing (so you know exactly what to approve): Each blocking dep is emitted as its own build log line and persisted to a structured build.blockedGrants array on the build record:

[blocked] relay [relay:send] — grant pending (grantId: a7d21c07-…). Approve at https://tawa.insureco.io/console/permissions/api-access

// build.blockedGrants[]
{ "service": "relay", "scopes": ["relay:send"], "status": "pending",
  "grantId": "a7d21c07-…", "approveUrl": ".../console/permissions/api-access",
  "reason": "Scope grant pending for relay [relay:send] (grantId: …) — approve at …" }

The Deploy blocked: … error reply also carries the grantId inline. Approve via the console, or PATCH /api/admin/scope-grants/{grantId} with { "status": "approved" } (super_admin or owner of the target service).

16. Module Registration (Bio-ID)

When: catalog.modules is non-empty.

What: Registers each module with Bio-ID. Returns a per-service API key injected as BIO_INTERNAL_KEY into the pod (scoped to this service, not the builder's master key).

Fail behavior: Non-blocking — logs warning if registration fails.

17. userAccess Provisioning (catalog 0.5.0+)

When: catalog.userAccess is declared.

What:

• Injects BIO_USERS_URL = https://bio.tawa.pro/api/v2/users
• Registers the user access scope in Koko (for tawa-web Console display)
• scope: cross-org → lists the service as an allowed consumer in the Koko record

Note: BIO_SERVICE_KEY is NOT currently injected by the builder — if your service needs one, provision it separately.

18. Helm Deploy

Command pattern:

helm upgrade --install {service} {chartPath} \
  --namespace {namespace} \
  --create-namespace \
  --set image.tag={imageTag} \
  --set image.repository={registry}/{service} \
  --set service.port={port} \
  --set health.endpoint={healthEndpoint} \
  --set secretRef={service}-managed-secrets \
  --set env.KEY=VALUE  ... (all provisionedEnvVars + service.config)
  # + Vault Agent annotations if Vault active
  # + NetworkPolicy labels for direct-transport deps

Helm chart priority:

1. .iec.yaml helmChart field
2. service.helmChart stored in builder DB (set via tawa services update --helm-chart)
3. Auto-discovered: helm/Chart.yaml, helm/{service}/Chart.yaml, helm/*/Chart.yaml, chart/Chart.yaml
4. Built-in helm/default-service chart (fallback)

Secrets: Managed secrets decrypted from service.secrets → K8s Secret {service}-managed-secrets → mounted via envFrom.secretRef.

NetworkPolicy: Pods default to receiving traffic only from tawa.pro/role: janus namespace (Janus sidecar). Direct-transport dependencies label the namespace so NetworkPolicy allows direct pod-to-pod traffic.

19. DNS Configuration

When: ENABLE_DNS_MANAGEMENT=true.

What: Creates or updates a Cloudflare CNAME record:

• {service}.{env}.tawa.pro (sandbox/uat) or {service}.tawa.pro (prod) → INGRESS_TARGET

Fail behavior: DNS errors are non-fatal — logged as [warn], deploy continues.

After DNS: registers domain binding in Koko.

20. Post-Deploy Pod Health Check

What: kubectl get pods -n {namespace} + recent events. Logged to build output. Stored as build.diagnostics if issues detected.

Fail behavior: Non-fatal — diagnostics only. Does not block subsequent steps.

21. Deploy-Gated Tests (catalog 0.5.0+)

When: IEC_TEST_URL is set AND catalog.tests is non-empty.

Target URL: Internal cluster URL (http://{service}.{namespace}.svc.cluster.local:{port}) — avoids external DNS propagation latency.

Flow:

1. POST {IEC_TEST_URL}/runs with test specs
2. Poll GET /runs/{id} every TEST_RUNNER_POLL_INTERVAL_MS (default 5s)
3. Timeout after TEST_RUNNER_MAX_POLL_MS (default 10 minutes)

Suites:

Fail behavior: Hard fail if tests fail. Deployment NOT rolled back.

22. Koko Service Registration

What: registerOrUpdateServiceInKoko — registers/updates the service in the service registry with:

• Routes (path, methods, auth, gas cost)
• Owner (org slug)
• Pod tier + storage tiers
• Direct-transport dependency labels (for NetworkPolicy)

23. Queue + Schedule Bindings

What: registerQueueBindings / registerScheduleBindings — tells Koko the upstream K8s URL for iec-queue and iec-cron to call when delivering jobs or firing schedules.

Fail behavior: Partial success is logged. Never blocks deploy.

24. Uptime Monitor (iec-pulse)

When: IEC_PULSE_URL is set.

What: Registers a monitor for https://{service}.{env}.tawa.pro{healthEndpoint}.

• prod: 30s poll interval
• sandbox/uat: 60s poll interval

Fail behavior: Fire-and-forget — never blocks.

Fail-Open vs Fail-Hard Quick Reference

Environment Variables

All builder env vars with their defaults and purposes:

Auto-Injected Platform Env Vars (Always Available in Pods)

These are injected by the builder for every deploy — services do NOT need to declare them in internalDependencies:

What Gets Written Where

Common Failure Patterns

"Deploy gate: insufficient gas reserve"

Org wallet balance is below the 3-month reserve. Run tawa wallet buy N to top up, then redeploy.

"Deploy gate: catalog spec.owner X does not match registered service owner Y"

Someone changed spec.owner in catalog-info.yaml. Either revert it, or use tawa services update <id> --org <new-org> if the transfer is intentional.

"Deploy blocked: Access to X has been denied/revoked"

The scope grant for a spec.dependencies entry was explicitly denied or revoked in Bio-ID. Check Console → Permissions → API Access.

"Deploy blocked: Scope grant pending for X [scopes] (grantId: …)"

A cross-org spec.dependencies entry created a grant request that nobody approved yet. The build log line [blocked] X [scopes] — grant pending (grantId: …) and build.blockedGrants[] name the exact grant. Approve it in Console → Permissions → API Access, or PATCH /api/admin/scope-grants/{grantId} { "status": "approved" } (super_admin / target-service owner), then redeploy. Note: while pending, {X}_URL is not injected — fix the grant, don't hardcode the URL.

"Convention check failed: N blocking violation(s)"

Source code violates a Koko gate rule (e.g., deprecated SDK usage). Fix the violations or use tawa deploy --skip-convention-check --reason "...".

"Config preflight failed: missing required config vars: KEY1, KEY2"

catalog.configDeclarations lists required vars that aren't set. Run tawa config set KEY1=... KEY2=....

"Undocumented upgrade: X Y → Z has no changelog entry"

An SDK was bumped without a Koko changelog entry. Run tawa changelog --sdk X to document it.

"Deploy-gated tests failed"

Post-deploy smoke tests failed. The pod is still running — use kubectl logs to debug. Re-deploy after fixing will re-run tests.

Pod stuck in CrashLoopBackOff after deploy

Check build.diagnostics in the builder DB or run kubectl describe pod in the namespace. Common causes: missing env vars, incorrect health endpoint, DB connection failure on startup.

Related: Builder Logs

The builder's own PM2 logs are redacted (write-time + read-time), rotated, and exposed through a platform-admin API. See Builder Logs — Redaction, Rotation & Admin API for the GET /platform/logs and GET /platform/logs/tail contracts.