Tawa Test

The Short Version

Tawa Test is the platform's test runner. One service — iec-test, at https://iec-test.tawa.pro — runs every check the platform makes against a service after it is deployed and before the build is called green. You declare what it should check in catalog-info.yaml; the builder relays the specs after helm upgrade and waits for the answer.

metadata:
  annotations:
    insureco.io/catalog-version: "0.7.0"   # 0.5.0 for spec.tests alone
spec:
  tests:
    smoke:
      - path: /health
        expect: 200
  walkthrough:                              # the standing screenshot tour
    title: "Collections"
    steps:
      - path: /collections
        expect: { h1: "Collections", "table tbody tr": "ACME" }

tawa walkthrough --prod            # run the tour on demand, no deploy
tawa walkthrough tour.yaml --prod --label pr-42

The feature set

Each suite is requested only when it has specs. A service that declares none is skipped entirely and the build log says Testing skipped.

Walkthroughs

A walkthrough is the feature the rest of Tawa Test exists to support: it drives a UI flow, asserts on every page, screenshots each step, and publishes an immutable page showing what shipped. A smoke check tells you a route returned 200. A walkthrough tells you what a person saw — and refuses to publish a picture of a screen whose assertion failed.

spec:
  walkthrough:
    title: "Collections & agency assignment"
    auth: portal-editor                                # a profile NAME, never a token
    viewport: { width: 1440, height: 900, scale: 2 }   # the default
    steps:
      - path: /collections/assign
        expect:
          h1: "Assign Collections"        # structural: the right page
          "table tbody tr": "ACME"        # substantive: the page has data
        note: "Assignment now writes the collector's agency (#46)."
        act: [{ click: "button:has-text('Scheduled reports')" }]
        shot: { selector: "[data-testid='panel']" }
        annotate: [{ at: "table thead th:last-child", text: "New agency column" }]
        redact: ["[data-pii]"]

What makes it evidence rather than decoration

1. Every step needs expect. A step without one is rejected by the catalog parser (the build fails before an image is built, naming steps[N]), by the builder's on-demand route, and by iec-test's own schema. There is no opt-out.
2. A failed step is never published as a clean screenshot. It renders as a failure card with the assertion table, expected vs. actual, and no image at all. The page generator ignores an image field on a failed record.
3. A failure says where the page actually was. element not found cannot distinguish a signed-out render from a wrong route from a 404, so every failure also records the step's url, the page's own <title> (failure.pageTitle) and the navigation's HTTP status (failure.httpStatus, null for a step that did not navigate). A 200 on the right URL under the title Sign in is a session problem; a 404 is a route problem. It is deliberately not a body excerpt — redact blurs pixels and cannot redact text, and a bundle is readable by anyone with its URL.
4. Assert on data, not chrome. h1 proves you are on the right page. It does not prove the page has content: a session can authenticate and still render an empty table.

The published bundle

Every run writes a directory, and the bundle is the page — CSS inlined, relative image references, no JavaScript, nothing fetched from the network. Copy it anywhere and it still opens.

walkthroughs/{service}/{environment}/{buildId}/
  ├── index.html        # the page: provenance header, per-step assertion table, overlays
  ├── manifest.json     # the machine-readable record — uploaded LAST, its presence means complete
  └── 01-collections-assign.png …

manifest.json is a versioned contract (version: 1). The fields worth knowing:

Bundles are immutable — a re-capture is a new buildId, and an upload onto an existing manifest is refused. A lost bundle is never a green gate: any upload failure turns the suite result into error.

Retention, per service and environment:

Running one on demand

A walkthrough normally runs as part of the deploy. To run one against the service as it is live right now — no build, no deploy, no gate — use tawa walkthrough:

tawa walkthrough --prod                      # the catalog's spec.walkthrough
tawa walkthrough tour.yaml --sandbox         # a spec from a YAML/JSON file
tawa walkthrough tour.yaml --prod --label pr-42 --open

The builder relays a one-off run with the walkthrough suite alone — smoke, e2e and compliance are not touched — under a synthetic build id, manual-{label-}{time}-{random}, so it never collides with a deploy's. Requires the member org role (viewer can read the result). Exit code is 1 when a step failed; an on-demand run is reported as failed in every environment, because "non-blocking in prod" is a property of the deploy gate, not of a run somebody asked for on purpose.

Which host a run drives

Both hostnames serve the same deployment, so every status check stays green either way — but an authenticated app's cookie host and login redirect are pinned to one of them. The session cookie is scoped to the host being driven, so driving the other renders signed out and fails every assertion with nothing saying why. The build log names both URLs.

Signed-in runs

auth: names a profile configured on iec-test — never a token, in a catalog or anywhere else. Two profiles matter on the platform:

A persona is the current answer and a staff-style profile is the role-less fallback. Full mechanics — how the token is minted, what the builder supplies, and why a persona can write — are in Walkthroughs → Auth and Test Personas.

Authoring captions locally

Captions live in the service's own catalog-info.yaml, which means writing prose about a screenshot you cannot see. The local loop fixes that — it runs spec.walkthrough against your dev server through the same runner and page generator the deploy gate uses, and re-runs on every save:

# in the iec-test repo
npx tsx src/cli.ts capture http://localhost:3000/login --out ./data/profiles/staff.json
npx tsx src/cli.ts walkthrough --catalog ../my-svc/catalog-info.yaml \
  --target http://localhost:3000 --local --watch --storage-state ./data/profiles/staff.json

Only the touched step and the rest of its page group re-run; untouched steps keep their PNG. A save that leaves the catalog invalid prints the exact path and keeps serving the last good bundle.

Smoke

HTTP probes. The one suite almost every service should declare, and the one whose failure stops every other suite in the run.

spec:
  tests:
    smoke:
      - path: /health
        expect: 200
        timeout: 5000       # optional

Entries carry path, expect and timeout — nothing else. An expectBody key on a catalog smoke entry is warned about and dropped: the status check runs, the body assertion never did. (The richer shape — method, headers, body, expectBody, expectHeaders — exists in iec-test's own spec format, reachable from the CLI and the HTTP API, not from a catalog.)

If smoke fails or errors, the orchestrator skips every remaining suite in that run.

E2E

Playwright browser flows — navigate, fill, click, assert, screenshot. Where a walkthrough asks "does this screen still show the right thing, and here is the picture", e2e asks "does this sequence still complete": log in, submit the form, land on the right page.

Runs in every environment. Blocking in sandbox and UAT; non-blocking in prod, the same posture walkthroughs have.

metadata:
  annotations:
    insureco.io/catalog-version: "0.5.0"
spec:
  tests:
    e2e:
      - name: "Login flow"
        auth: portal-editor              # a profile NAME, never a token
        timeout: 30000                   # optional, ms
        steps:
          - { action: navigate, url: /login }
          - { action: fill, selector: "#email", value: [email protected] }
          - { action: click, selector: "button[type='submit']" }
          - { action: wait, selector: "[data-testid='dashboard']", state: visible }
          - { action: assert_url, pattern: /dashboard }
          - { action: assert_text, selector: h1, text: Dashboard }

The step vocabulary

{{SERVICE_URL}} and {{RUN_ID}} interpolate in url and value.

There is no evaluate. Running arbitrary JS in a browser on iec-test's pod network is refused from any remote caller; it exists only in the local CLI. A catalog that declares it fails the build at parse time, by name.

Steps in a file instead

A long flow does not have to live in the catalog. Point at a repo file holding { steps: [...] }:

spec:
  tests:
    e2e:
      - name: "Collections"
        spec: e2e/collections.yaml     # relative to catalog-info.yaml

# e2e/collections.yaml
steps:
  - { action: navigate, url: /collections }
  - { action: assert_text, selector: "table tbody tr", text: ACME }

.yaml, .yml or .json. The path is relative to catalog-info.yaml and may not be absolute or contain .. — the builder loads it out of the clone and will not read outside it. An entry declares either steps or spec, never both and never neither.

Everything is checked before the image is built

The builder validates every step against the vocabulary above at catalog-parse time, and names what it found:

Catalog "my-api": spec.tests.e2e[2].steps[1] (action "click") requires "selector" (a non-empty string).
Catalog "my-api": spec.tests.e2e[0].steps[0] has unknown action "type". Valid actions: assert_text, assert_url, …

An unknown key on a step is an error rather than a warning: a misspelt selector silently dropped leaves an assertion that cannot fail. tawa preflight (0.27.0+) runs the same checks without spending a build.

Signed-in flows

auth: names the same profiles walkthroughs use, resolved by iec-test from its own configuration — see Signed-in runs above and Test Personas. Without it the flow runs signed out, which for a logged-in app means asserting on a login page. A profile that cannot be established fails the flow rather than quietly running it anonymously.

Which host it drives — the same caveat as a walkthrough

A deploy-time e2e run drives the cluster-internal URL. An app whose session is bound to its public host renders signed out there through no fault of the deploy (iec-test#39) — which is exactly why a prod failure is non-blocking rather than a gate. Relative navigate urls resolve against whichever host the run is driving, so the flow itself needs no change between the two.

Compliance

Asks Septor and the wallet whether the events a service claims to emit are really there, with retries because those writes are asynchronous.

spec:
  tests:
    compliance:
      septor:
        - eventType: payment.created
          entityId: "{{orgSlug}}"
      wallet:
        - type: transaction
          walletId: "wallet-{{orgSlug}}"

A check that cannot be performed — missing config, a transport failure — is reported as error, never as fail, so the gate neither passes nor fails on an infrastructure problem.

Moving a spec.compliance block under spec.tests turns it on. In its correct position it becomes a deploy gate that queries Septor on every build, and an environment that has never emitted those events fails every deploy until it has real history. Move it once the service has produced the events you are asserting on.

Integration

Builds each declared dependency's cluster-internal URL and probes it. It is a real suite in iec-test and it is accepted by POST /runs — but there is no spec.tests.integration, so a catalog cannot ask for it. Drive it from the CLI:

npx tsx src/cli.ts run specs/deps.json --target https://my-api.tawa.pro \
  --suite integration --environment prod

Gating, per environment

The prod exemption covers the two browser suites, and it is narrow: it applies only when they are the only failing suites. A run where smoke also failed fails the build as usual.

Both are driven at the cluster-internal URL, where an app whose session is bound to its public host renders signed out through no fault of the deploy — failing a prod deploy on that would teach people to delete the suite rather than fix anything.

e2e used to be skipped in prod entirely, which meant it never ran: nothing on this platform deploys to sandbox or UAT. Fixed in iec-builder#161 along with the payload shape (see E2E).

A failed gate does not roll back the deployment. The pod stays up; the build is marked failed. Fix and redeploy.

Reading the result

• The build log — Test result: walkthrough: passed (5/5), then Walkthrough: <url>.
• The page — https://iec-test.tawa.pro/walkthroughs/{service}/{env}/{buildId}/index.html. Provenance header (service, environment, build id, commit, capture time, viewport, auth profile, N of M steps passed), each step with its assertion rendered verbatim, positioned annotation overlays, redaction notes, and a ← Previous build link.
• manifest.json beside it — the fields in the table above.
• The index. https://iec-test.tawa.pro/ui/walkthroughs lists every walkthrough still in storage, by service and then newest first, with search by label or title. It needs a Bio-ID sign-in (org insureco or super_admin); the bundle URLs themselves stay open by URL. See Walkthroughs → Finding a walkthrough later.
• The status page. When the deploy-gated tests pass, the builder attaches the bundle URL to the service's changelog in Koko, so the latest walkthrough is linked from the What's new panel on the service's status page. A redeploy of the same commit that produces no new notes still gets its walkthrough link attached to the changelog Koko already holds. See Uptime Monitoring.

Bundle pages are served by iec-test itself as a deliberately dumb read-through: the filename must be one the bundle's own manifest lists, content types are fixed by filename, there is no listing, and nothing from the request becomes part of an object key before it has matched.

Where the code and these docs disagree

Recorded here because the code is what runs.

There are five suites, not four. integration is registered in iec-test alongside smoke, e2e, compliance and walkthrough, and POST /runs accepts it — the builder simply never asks for it, because spec.tests has no key for it.

Execution order is fixed by the runner registry, not by the order you declare: smoke → integration → e2e → walkthrough → compliance. (This page previously listed compliance before walkthrough; RUNNERS in src/runners/registry.ts has the walkthrough fourth.) If smoke fails or errors, every remaining suite is skipped.

What NOT to do

# ❌ a walkthrough step with no assertion — rejected everywhere, at parse time
- path: /reports
  note: "The new reports page"

# ❌ chrome only, on a data page — goes green on an empty table
- path: /collections
  expect: { h1: "Collections" }

# ❌ a token where a profile name belongs
auth: eyJhbGciOi...

# ❌ expectBody on a catalog smoke entry — warned and dropped, the assertion never runs
spec:
  tests:
    smoke:
      - path: /api/health
        expect: 200
        expectBody: { success: true }

# ❌ spec.compliance at the top level — parses, deploys, does nothing
spec:
  compliance:
    septor: [...]

# ❌ an e2e step with no assertion — a flow that only navigates proves nothing
spec:
  tests:
    e2e:
      - name: "Dashboard"
        steps:
          - { action: navigate, url: /dashboard }

# ❌ a hostname in an e2e navigate — a deploy-time run drives the cluster-internal
#    URL and an on-demand one drives the public URL, so one of them is always wrong
- { action: navigate, url: "https://my-api.tawa.pro/login" }

# ❌ evaluate in a catalog — refused at parse time; it is CLI-only
- { action: evaluate, script: "window.localStorage.setItem('x','1')" }

# ✅
spec:
  tests:
    smoke:
      - path: /health
        expect: 200
    e2e:
      - name: "Login flow"
        auth: portal-editor
        steps:
          - { action: navigate, url: /login }
          - { action: fill, selector: "#email", value: [email protected] }
          - { action: click, selector: "button[type='submit']" }
          - { action: assert_text, selector: h1, text: Dashboard }
    compliance:
      septor:
        - eventType: payment.created
          entityId: "{{orgSlug}}"
  walkthrough:
    title: "Collections"
    auth: portal-editor
    steps:
      - path: /collections
        expect: { h1: "Collections", "table tbody tr": "ACME" }
        redact: ["td.insured-name", "td.email"]

• Don't put a one-off feature walkthrough in the catalog — it runs on every deploy from then on. Use tawa walkthrough <spec> for that.
• Don't treat a deploy-time walkthrough of an app whose session is bound to its public host as reliable — it drives the cluster-internal URL. Run that one on demand.
• Don't expect a failed gate to roll anything back. It doesn't.
• Don't publish bundle ids where they invite enumeration. Anyone with a URL can read that bundle; redact is the primary control and the id namespace is the second. There is no third.

See also

• Walkthroughs — the spec in full, the authoring rules, and auth.
• Test Personas — a logged-in session without a password.
• spec.walkthrough in the catalog reference
• Build & Deploy Pipeline — step 21, where the gate runs.
• Uptime Monitoring — the changelog panel the bundle link lands in.