--- paths: - ".github/workflows/*.{yml,yaml}" - ".github/actions/**/*.{yml,yaml}" --- # CI / GitHub Actions ## Conventions - **Runners**: runs-on.com ephemeral Graviton fleets. Shapes live in [`CFXLabsInc/cfx-self-hosted-services`](https://github.com/CFXLabsInc/cfx-self-hosted-services/blob/main/projects/runs-on-fleet/catalog.tf) at `projects/runs-on-fleet/catalog.tf` — that file is the only place a shape is defined. Target one with `runs-on: runs-on/fleet=/env=production`. There is no `.github/runs-on.yml` in this repo any more: it held nothing but an `_extends` on `CFXLabsInc/.github-private`, which was deleted, so it was removed rather than left pointing at a repo that is gone. - **There are exactly two fleets, `small` and `medium`, and both pin the `d` subfamilies.** Every shape states `cpu` and `ram` explicitly, but **the declared range is not what a job gets** — spot takes the cheapest qualifying shape every time. Size against what it actually draws: | fleet | declared | **actually drawn** | for | | -------- | -------------------- | ---------------------------------- | ---------------------------------------------------------------------------------------------------- | | `small` | 2-4 vCPU / 4-16 GiB | `*.large` — 2 vCPU / **3.7 GiB** | orchestration: detect changes/stacks, dbmate, pulumi-only previews, publish-specs, actionlint | | `medium` | 4-8 vCPU / 16-32 GiB | `*.xlarge` — 4 vCPU / **15.3 GiB** | everything else: build, typecheck, test, lambda/container CD, SST deploys, lint, typedoc, `app-test` | Measured over 60 `medium` and 84 `small` draws — `medium` never once drew an 8-vCPU box. The top of a `ram` range is never insurance. - **`family` is matched by PREFIX.** `["c7g", "m7g"]` also admits `c7gd`, `c7gn`, `m8gn` — which is how `small` spent months drawing NVMe shapes 47 times out of 84 while its comment claimed it had none. Spell subfamilies out when you mean them. - **The `d` pin is what puts the workspace on local NVMe.** RunsOn only overlays `/home/runner`, `/tmp` and `/var/lib/docker` onto `/mnt/ephemeral` when an instance store exists; without it they stay on the `gp3:30gb` root, which starts a job with ≈11 GiB free. Every job runs `bun install`, so both fleets need it. - **There is no `large` fleet.** It was 8 vCPU / 16-64 GiB with `ecr-cache`, removed in [cfx-self-hosted-services#99](https://github.com/CFXLabsInc/cfx-self-hosted-services/pull/99) because measurement showed nothing needed it: every job that targeted it peaked under `medium`'s 15.3 GiB (worst, admin-dashboard's SST build, 12.6 GiB). It had already been drawing 15.3 GiB boxes part of the time and those runs peaked **lower** — 9.5 GiB for the same build — because V8's default heap is memory-aware and collects sooner on a smaller box. **Demand adapts to the shape; it does not overflow.** Adding it back means restoring the `runners` and `fleets` entries in `catalog.tf` **and** the label in this repo's `.github/actionlint.yaml`, which is deliberately absent. - **`ecr-cache` went with it and was the wrong tool anyway.** It provisions an _ephemeral, per-run_ registry, so it cannot hold cross-run cache; that lives in the per-app workload-account `cfx--buildcache` ECR repo. `enable_ecr = true` is now orphaned in `main.tf`. - **The Test jobs' `--parallel=3` cap is a leftover, not load-bearing.** Both Test jobs once peaked at 19-21 GiB and the runner was reaped four times, because the suite backed its database with **in-process PGlite that every worker carried its own copy of**. [#3091](https://github.com/CFXLabsInc/cfx-platform/pull/3091) moved it to a shared Postgres container: peak fell to ≈4.9 GiB at the same wall time and cap, and nothing has been reaped since 2026-08-08. The cap stays as cheap insurance — it binds only when 3+ container-heavy legs are ready at once — but **it is no longer what keeps `main` green, and a bigger fleet was never the fix.** Keep the two jobs in lockstep, and keep the cap on the step, never in `nx.json`. - **Identical victims across runs is how you tell a reap from a spot interruption.** The 2026-08-08 pair killed the same five legs — `db`, `bank-services`, `customer-services`, `entity-services`, `token-services`, all container-backed — on different instances 33 minutes apart. Spot reclamation is random and would not pick the same five names twice. Count the `##[group]❌` lines: if the count equals the effective `--parallel`, the tasks were reaped mid-flight, not failed. - **A red `main` compounds.** Its affected set is computed against the last successful `main` run (nx-set-shas), so every failure widens the next window into a 100+ task run that `pr.yml`, which diffs against the PR merge-base, never sees. That asymmetry is why this reproduced in CI and not locally. - **There are no `deploy` / `deploy-large` fleets, and no private fleets at all.** Both existed only so a job could route to the workload VPCs. Runners now join Cloudflare Zero Trust at boot and reach Aurora and Valkey **by hostname from any fleet** over a WARP tunnel baked into the AMI ([cfx-self-hosted-services#87](https://github.com/CFXLabsInc/cfx-self-hosted-services/pull/87), [#89](https://github.com/CFXLabsInc/cfx-self-hosted-services/pull/89)), so both were deleted rather than kept as aliases. `deploy` and `deploy-large` → `medium`, or `small` for pure orchestration. - **Size the job, not the network.** admin-dashboard's SST deploy prerenders against Aurora + Valkey, but that is a _reachability_ need and WARP covers it from anywhere. Put it on `small` and you get the no-job-log OOM signature below, which reads as a network problem and is not one. - **Never set a `NODE_OPTIONS` heap ceiling on a runner.** `--max-old-space-size=10240` rode along from the Vercel era on both SST jobs and was removed in [#3135](https://github.com/CFXLabsInc/cfx-platform/pull/3135). On a ≈16 GiB box it is worse than useless: it lets V8's old space reach 10 GiB before collecting hard, so with the OS, the runner agent and OpenNext's children also resident the heap can exceed physical memory and the kernel reaps **the agent** — the no-job-log signature below, which looks nothing like a memory ceiling you set yourself. Node's default is memory-aware and sizes itself to the shape it lands on. (Removing it changed measured peak by nothing, within ±0.35 GiB — the ceiling was never binding. It was a latent hazard, not a live cost.) - **`preinstall` does NOT wait for the WARP tunnel — jobs that need it wait for themselves.** Blocking there cost a p50 of 6.1s and a p90 of 10.2s on _every_ job in the org, the second-largest startup phase after `agent-booting`. It is genuine connect time (the unit needs ≈26s from boot; `agent-booting` covers ≈20s), so waiting does not make it faster — it just charges the ≈95% of jobs that never resolve an in-VPC hostname. A job that dials Aurora or Valkey adds this immediately before that step, where checkout + `bun install` have already absorbed the wait and it returns instantly: ```yaml - name: Wait for the WARP tunnel run: systemctl is-active --wait cfx-warp.service || true ``` Current consumers: `cd-dbmate-aurora.yml`, both `Flush caches` jobs and `cfx-db` dev/prod in `deploy-env.yml`, `adhoc-pulumi-refresh.yml`, `cd-sst-nextjs.yml`, and `pr.yml`'s `sst-preview`. **Add a new in-VPC step to that list or it races.** - **Peak memory and CPU for every job are already in its log.** The AMI's `/runs-on/post.sh` hook prints a `📊 Job Metrics` block at cleanup with `Memory Utilization (%) (min/max/avg)`; multiply by `InstanceRAM` from the `🏃 Runner details` table at job start to get GiB. It is system-wide, not per-process. **A missing metrics block is itself the signal** — a reaped runner never reaches the hook, so the jobs you most need data for are the ones with none. **Measure before resizing anything.** - **That block is free, and it is the richest one. Do NOT declare `metrics:` on `runs-on/action@v2` to get it.** The action's `metrics:` input is a second, independent collector: it runs the CloudWatch agent and publishes to the `CWAgent` namespace, billed **per metric per month** — RunsOn's own docs call CloudWatch "the more expensive surface for high-cardinality runners", and ephemeral runners are the definition of high cardinality. Every job in this repo declared `metrics: cpu,memory,disk`; at ≈2,850 live metric streams in any 3-hour window that reached **$218/mo, more than the entire EC2 spend for the fleet** ($223/mo). It was removed from all 48 job definitions. - **Nothing was lost.** The `runs-on/action@v2` step itself stays (it starts the S3 Magic Cache sidecar), only the input is gone. The CloudWatch block reports total CPU, memory and disk; `/runs-on/post.sh` reports **per-core** CPU, load average, memory, disk I/O, network I/O, filesystem utilisation, and a `✅ No significant resource constraints detected` verdict. Proof: `actionlint` in `pr.yml` has no `runs-on/action` step at all and still prints the full block. - Its stated purpose was durability past the log's retention, but the series are dimensioned by **ephemeral `InstanceId`** — they cannot be grouped or trended by job name, so what outlived the log was unqueryable. If durability is ever genuinely needed, use the OTLP surface, not CloudWatch. - **In-VPC reachability now fails differently.** It is not a placement mistake any more, it is a tunnel failure, and it surfaces mid-job as `getaddrinfo ENOTFOUND` rather than as a connection timeout. Every job logs its tunnel state during `Set up runner` — **read that line before debugging a database error**: ``` preinstall: WARP connected — in-VPC hostnames resolvable preinstall: WARP NOT connected — in-VPC hostnames will NOT resolve ``` The service fails open on purpose (a Cloudflare outage must not stop the ≈95% of jobs that never touch the VPC), so a broken tunnel produces a runner that starts normally and a job that dies later. - **There are no x86 fleets, and no `medium-docker`.** `medium-x86` / `large-x86` were defined but nothing ever targeted them, and they were deleted. Nothing anywhere targets x86 on RunsOn today: the org-wide profiles that once declared those labels lived in `CFXLabsInc/.github-private`, which is gone, and cfx-solana-program-library — the repo usually cited for this — runs `ubuntu-24.04` for everything except one `runs-on/fleet=medium/env=production` release job. The container workflows default to `medium`. A new fleet must be added to `catalog.tf` (both the `runners` and `fleets` maps) **and** to every consumer repo's `.github/actionlint.yaml` — actionlint is per-repo and cannot see the org-wide catalog. All shapes enable Magic Cache (`s3-cache`) — in-VPC S3 actions/cache — and both pin the `d` subfamily, so local NVMe is guaranteed everywhere. **RunsOn does the NVMe rebase itself** — its `runner-setup-mount-points` step overlay-mounts `/home/runner` and `/tmp` onto `/mnt/ephemeral` before `preinstall` runs, so workspace, action \_work and per-user caches land on local SSD instead of the `gp3:30gb` root EBS. `/var/lib/docker` is mounted by the same step. The per-runner `preinstall` used to do this with `rsync` + `mount --bind`, and used to restart dockerd afterwards; both were removed once RunsOn's own handling made them redundant. `preinstall` is now a WARP status line plus a `df`/`mount` diagnostic and nothing else (single-digit ms, down from 19.7s). The `gp3:30gb` root cannot be removed — it is the boot device, and the instance store is overlaid **onto** `/home/runner`, `/tmp` and `/var/lib/docker` rather than replacing `/`. It was shrunk from 40 GB in [cfx-self-hosted-services#101](https://github.com/CFXLabsInc/cfx-self-hosted-services/pull/101) and cannot go lower: measured usage is ≈17.7 GiB and 30 GB is the base AMI's own size. **Move Packer first if it ever shrinks again** — a block device cannot shrink below its snapshot, so a fleet asking for less than the AMI is a rejected launch, not a warning. - **Every root volume is provisioned at 400 MB/s, against a 125 MB/s free baseline**, which bills the 275 MB/s difference at $0.040/MB/s-month on every runner *and* every warm-pool instance — ≈$124/mo, second only to the metrics above. The root holds the OS and nothing else (see the overlay note), so it does not need the throughput. `volume` takes it: `"gp3:30gb:125mbs:3000iops"`. Measure instance boot time before and after — AMI hydration reads from EBS, so this trades a standing charge against a possible startup delay. Cross-run buildkit cache has one home: the per-project `cfx--buildcache` ECR repo (via `image-build-push`'s `--cache-to=type=registry`). There is no sticky disk on any fleet. - **Two runner-death signatures, two causes.** Neither is a step failure, so neither has a useful step log: - `The runner has received a shutdown signal` — spot interruption, or the kernel reaping a process mid-job. - `The self-hosted runner lost communication with the server` **with no job log at all** (`gh api .../jobs//logs` → `BlobNotFound`, steps have empty conclusions, post-steps never ran) — the machine died under the agent. Almost always undersized: the job's memory ceiling exceeds the fleet's `ram`. Read it from `gh api repos///check-runs//annotations`; it is written nowhere else. - **Setup**: every job inlines its own toolchain — there is no shared setup composite in this repo's workflows. The block is two standard steps, in this order: ```yaml - uses: runs-on/action@v2 - name: Install dependencies env: NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }} run: bun install --frozen-lockfile - uses: nrwl/nx-set-shas@v5 # only in jobs that run `nx affected` / `nx format` ``` - **Check out with `filter: blob:none` + `persist-credentials: false`, never `filter: tree:0`.** Both are partial clones; the difference is that `tree:0` omits trees as well as blobs, so the first `git diff --name-only` re-fetches every tree it needs one lazy round trip at a time. That diff is exactly what `nx affected` (between `NX_BASE`/`NX_HEAD`) and `dorny/paths-filter` do, and nothing errors when it goes slow — the job just quietly pays network for objects it declined to clone. `blob:none` keeps all trees and omits only file contents, which is the shape those tools want. Reserve `tree:0` for jobs that read commit metadata and never a file list. `persist-credentials: false` keeps `GITHUB_TOKEN` out of `.git/config` where any later step can read it; drop it only on a job that actually pushes over git (none do today). - **`blob:none` + `persist-credentials: false` means no lazy blob fetch — any git command that reads file CONTENT fails.** The partial clone defers blobs to the promisor remote, and without persisted credentials that fetch cannot authenticate: `fatal: could not read Username for 'https://github.com'` followed by `could not fetch from promisor remote`. The trap is that the offending command usually looks name-only: `git diff --name-only` still runs **rename detection** by default (`diff.renames` is on since Git 2.9), and that reads blobs. Pass `--no-renames` — comparing names needs only trees, which the clone already has. Same applies to `git log -p`, `git grep` over history, and `git show :`. - **`fetch-depth: 0` only where history is read** — `nrwl/nx-set-shas` walks history for the last successful SHA, so it needs the full clone. Jobs that don't should stay on the default `1`. - **At `fetch-depth: 0` the remote refs are already there — never add a `git fetch origin main` after checkout.** Depth 0 fetches `+refs/heads/*:refs/remotes/origin/*` (`actions/checkout`'s `getRefSpecForAllHistory`), so `origin/main` lands with the clone and a three-dot range like `origin/main...HEAD` resolves with no extra step. A follow-up fetch doesn't merely duplicate that work, it fails the job: `persist-credentials: false` keeps `GITHUB_TOKEN` out of `.git/config` and this repo is private, so the fetch dies on `fatal: could not read Username for 'https://github.com'`. - **`actions/cache` restores work.** They did not until `s3:ListBucket` was added to `cfx-workload-boundary` in the RunsOn account (`aft-account-customizations` `1cf0d3c`, #74) — the fleet instance role carries that boundary, so a missing action there denied every read while writes still succeeded, making caches save-only. Symptom was a ≈30s retry of `GetCacheEntryDownloadURL` before each miss, because the sidecar returns the raw AWS error string where the action expects JSON. History in [#2854](https://github.com/CFXLabsInc/cfx-platform/pull/2854). If cache reads ever go silently cold again, check that boundary first. - **Use plain `actions/cache`**, per [RunsOn's guidance](https://runs-on.com/docs/performance/caching/actions/): _"For RunsOn workflows, prefer the Magic Cache (`extras=s3-cache`) — it requires no per-step changes… Reach for `runs-on/cache` only when you need the standalone action behavior, for example caching from runners outside RunsOn."_ Don't swap in `runs-on/cache`; it goes to the same bucket under the same role. - **Cache `~/.pulumi/plugins` AND `~/.pulumi/dynamic_tf_plugins` in every job that runs `pulumi`, keyed on `bun.lock` plus `packages/pulumi/sdks/*/package.json`.** Pulumi acquires every provider whose `PulumiPlugin.json` names a `github://` server from the GitHub releases API — that is **all** of them, first-party included, at two calls each (release lookup + asset download). Unauthenticated that is 60 req/hr per source IP, and the whole runs-on fleet shares one NAT, so this repo's ≈44 pulumi jobs drain it in minutes and every stack fails with `rate limit exceeded: 403 … Could not automatically download and install resource plugin`. `GITHUB_TOKEN` on the `pulumi/actions` step lifts the ceiling to 1,000 req/hr per repository; the cache keeps the steady state near zero so that ceiling is never binding. **Keep both** — the token covers cold runners and newly-added providers, the cache covers everything else. - **`dynamic_tf_plugins` is a second, separate acquisition path with NO token knob at all — and it is not rate limiting that breaks it.** The bridged SDKs in `packages/pulumi/sdks` (sentry, doppler, hookdeck, temporalcloud, propelauth) are `pulumi package add terraform-provider` output: the Pulumi plugin is `terraform-provider` v1.0.1, and it resolves the real Terraform provider from `registry.opentofu.org` **at runtime**, caching it under `$PULUMI_HOME/dynamic_tf_plugins` (override: `PULUMI_DYNAMIC_TF_PLUGIN_CACHE_DIR`). That resolution reads the provider's authentication checksums off github.com, and the embedded OpenTofu registry client in the plugin binary supports no credential for it — no `TF_TOKEN_*`, no `TF_CLI_CONFIG_FILE`, no `provider_installation` block. `GITHUB_TOKEN` does not reach it and never will, so `504 Gateway Timeout returned from github.com` fails the whole stack (`pulumi:providers:sentry **failed** 1 error`) with nothing to authenticate away. Caching the directory is the only lever: a warm run never makes the request. SST stacks are unaffected — they use `@pulumiverse/sentry`, a native Pulumi provider that lives in `plugins`. - Keyed on the lockfile because that is what pins provider versions: it changes rarely, so the ≈800 MB re-save is paid on dependency bumps rather than every run, and the prefix `restore-keys` means a bump still restores the previous set and downloads only what moved. **The SDK manifests are in the key because `bun.lock` cannot see them** — it records each bridged SDK as a bare `file:packages/pulumi/sdks/` spec, so bumping the pinned Terraform provider version inside one leaves the lockfile hash identical and the cache would never re-save. - **This is the one large archive the repo keeps**, and it contradicts the no-`node_modules` rule on purpose: there is no in-VPC mirror for Pulumi plugins, so the alternative is a rate-limited fetch over the public internet, not a fast local one. If it ever measures net-negative, drop the cache step and keep `GITHUB_TOKEN` — that alone unblocks the Pulumi providers. It does not unblock the bridged Terraform ones, so `dynamic_tf_plugins` stays either way. - Must come **after** `runs-on/action@v2`, whose sidecar keeps the traffic in-VPC. - **Don't cache Playwright browsers — but do keep the `playwright install` step.** The AMI preinstalls chromium plus its apt deps and exports `PLAYWRIGHT_BROWSERS_PATH=/opt/ms-playwright`, so `~/.cache/ms-playwright` is never written and a cache step on that path archives an empty directory. The install step is a different question and must stay: Playwright never downloads a missing browser lazily — `browserType.launch` hard-errors `Executable doesn't exist at …` — so deleting it turns any browser/CLI skew into a red build instead of a slow one. Note headless mode needs `chromium_headless_shell-`, a **separate** download from headed `chromium-` since 1.49; both revisions are implied by the CLI version. - **What makes that install cheap is an exact pin.** `@playwright/test` in the root `package.json` must be an **exact** version equal to `PLAYWRIGHT_VERSION` in `cfx-self-hosted-services/packer/runs-on-runner/scripts/preinstall-playwright.sh` (both `1.62.0`) — then the install short-circuits to a checksum check. A caret range is enough drift to make every e2e run evict the baked build and refetch ≈500 MB. `packages/admin-dashboard/package.json` declares it too (`@nx/dependency-checks` owns that entry) and must carry the same exact pin. Bump all of them together, never one alone. - **`sticky_cache: git` is INCOMPATIBLE with this repo's checkout. Measured ≈10x slower, reverted — do not re-add it without changing the checkout first.** Same commit, same run ([31557848164](https://github.com/CFXLabsInc/cfx-platform/actions/runs/31557848164)), wired jobs against unwired ones: `Setup` checkout **95s vs 7s**, `Lint` 98s, `Test` 74s, `Build + Specs` 71s — while `Publish Specs`, `Publish Docs` and every `Deploy Dev` leg stayed at 6-8s. It does not amortize; the later run was worse. - **Why, precisely.** `filter: blob:none` makes a blobless partial clone, so git must lazily fetch every file's blob from the **promisor remote** to populate the working tree. GitHub serves those as batched fetches. The `git` mode starts a local proxy and rewrites `https://github.com/` with a global `url.insteadOf`, so the promisor remote becomes the proxy — and RunsOn documents that "rewritten URLs no longer match the credentials configured by `actions/checkout`" for private repos. Combined with `persist-credentials: false` (see the promisor trap above) and the known partial-clone behaviour of fetching blobs one at a time when a tree has many files, the lazy fetch degrades into a per-blob crawl. The mirror is not the problem; the interaction with partial clone is. - **So it is not a tuning question.** Making it pay would mean dropping `filter: blob:none` (giving up the partial clone this repo deliberately uses) or feeding the proxy a PAT. Checkout is 7s today. There is nothing to win. - `buildkit` mode in `container-build-push` is **kept** — persistent builder state is a real win and never touches checkout. - If anyone re-tests `git` mode: it must run BEFORE `actions/checkout`, and you must compare a wired job against an unwired one **in the same run**. Comparing across runs hides it, because a cold `Setup` job on `fetch-depth: 0` looks slow for unrelated reasons. - **`buildkit` mode requires `cleanup: false`** on `docker/setup-buildx-action`, so RunsOn can stop the builder before the disk is snapshotted; buildx's default teardown would destroy exactly the state being persisted. Order is fixed: `runs-on/action` → `setup-buildx-action` → `build-push-action`. - **Don't pass `buildkitd-config-inline`.** With `ecr-pull-through` the runner agent writes `~/.docker/buildx/buildkitd.default.toml` before the job and `setup-buildx-action` discovers it, so the builder gets the ECR Docker Hub mirror for free. An inline config overrides that file. - **`runs-on/action@v2` is the very first step of every job, above `actions/checkout`** — it starts the S3 Magic Cache sidecar, so nothing that reads or writes the cache (checkout's own fetch included) should precede it. It is also still required alongside the `s3-cache` extra for `actions/upload-artifact`. No-op off RunsOn, so never guard it with an `if:` that depends on a later step's outputs — that reference is empty at step 1 and silently skips the sidecar for the whole job. - **There is no `oven-sh/setup-bun` step and nothing about bun is cached.** bun is preinstalled on the RunsOn AMI, so CI uses the ambient binary — the action only ever downloaded (and tool-cached) a second copy. `~/.bun/install/cache` is not cached either: a cold `bun install --frozen-lockfile` is fast enough that the archive round-trip cost more than it saved, and that cache carried real footguns: `node_modules` was never cacheable (bun links it isolated, so the tree is not relocatable — [oven-sh/bun#18752](https://github.com/oven-sh/bun/discussions/18752)), and the store key had to be **arch-scoped** because bun extracts only the host platform's optional deps, so an arm64 store restored onto an x86 runner hands it the wrong native binaries (the same arch sensitivity called out for lambdas below). - **`NODE_AUTH_TOKEN` is the whole auth story.** bun reads `_authToken` only from the **project** `.npmrc`, never `$HOME/.npmrc` — the committed `.npmrc` + `bunfig.toml` expand `${NODE_AUTH_TOKEN}` at install time. A token written to `$HOME/.npmrc` (what `setup-node`/`aws codeartifact login` do) never reaches bun's tarball fetch. Requires `packages: read` on the job. - **`.bun-version` no longer binds this repo's CI** — nothing reads it now that `setup-bun` is gone, so the version CI runs is whatever the AMI ships. Keep the file anyway: it is still the local/tooling pin, and it must not move to `package.json#packageManager` (a bun `packageManager` field would break the corepack pnpm shim that cfx-demo / cfx-marketing still rely on). `BUN_VERSION` in `docker/Dockerfile.node` is the pin for images and is unaffected; the AMI's bun is now a third, independent version — bump it in `cfx-self-hosted-services` when CI needs a newer one. - **`nrwl/nx-set-shas@v5` only where affected semantics apply** — `nx affected` and `nx format:check` (which scopes by `NX_BASE`/`NX_HEAD`). A job running a fully-qualified `nx run :` doesn't need it, and on `schedule`/`workflow_dispatch` triggers with no prior successful run it just logs a warning and falls back to `HEAD~1`. Needs `actions: read`. - **No CodeArtifact login.** The in-VPC `cfx/npm` pull-through is reachable only through `$HOME/.npmrc`, which `bun install` ignores, so `aws codeartifact login` bought this repo nothing while rewriting the default registry for every other tool in the job (that is what `NPM_REGISTRY` in `cron-sst-preview-reaper.yml` exists to undo). The reusable workflows used to run it via the `setup-nx` composite; that composite is gone and every one of them now inlines the same bun toolchain, so no workflow here calls `aws codeartifact login` any more. - **Runner arch is load-bearing for lambdas**: bun installs only the host platform's optional deps (pnpm's `supportedArchitectures` has no bun equivalent), so lambda bundles emit the single `linux--gnu` `@valkey/valkey-glide` binary named by `LAMBDA_TARGET_ARCH` in `esbuild/lambda.config.mjs`. That constant must track both the arm64 runners **and** `architecture` in every `packages/*/pulumi/Pulumi.*.yaml`; moving a lambda to x86_64 without moving its build runner is a hard build failure, by design. - **Nx task caching is Nx Cloud's; nothing on disk is cached.** `nxCloudId` in `nx.json` is the whole client config — there is no `actions/cache` step on `.nx` anywhere, and `.nx` is fully gitignored. - **`neverConnectToCloud` must stay absent.** `isNxCloudDisabled()` short-circuits `isNxCloudUsed()`, so the key silently reverts every job to the local `DbCache` while leaving `nxCloudId` in place looking correct. Same for `NX_NO_CLOUD=true`. - **`cacheDirectory` and `maxCacheSize` are gone too.** Both only ever bound the local `DbCache`; retention is Nx Cloud's now. - **`NX_SKIP_REMOTE_CACHE=true` in `docker/Dockerfile.node`** keeps image builds off the cloud — that stage holds no credentials and gains nothing from a network round-trip. - **Two mechanisms were removed here, not one.** `@nx/shared-fs-cache` (activated by a committed `.nx/key/key.ini`, deprecated, CVE-2025-36852) served CI, and 24 whole-directory `actions/cache` steps shipped it between jobs. That archive never evicted — the `nx-test` entry on `main` grew 0.18 → 13.15 GiB in 8 days and uploaded ≈483 GiB to carry 13 GiB, 36.7x write amplification. The earlier `@nx/s3-cache` remote cache (bucket + `cfx-nx-cache-s3-user`) is gone as well. - **Leftover `-nx-` archives still sit in the RunsOn bucket** until `cache_expiration_days` (**10**, in `cfx-self-hosted-services/projects/runs-on-fleet/main.tf`) drops them. Nothing reads them. That value was 1 only because the `.nx` archives were 2.26 TiB of a 2.36 TiB bucket; with Nx Cloud owning that cache the remainder is the bun store and Pulumi plugins, which a daily sweep only made slower. - **Don't pass `--output-style`.** `shouldUseDynamicOutput` returns false under `isCI()`, so CI is already `static` — each task's output under its own header, with a summary. The Test job carried `--output-style=stream` while a failure's full log lived nowhere but the runner; it interleaved five parallel tasks into one unattributable wall. Nx Cloud has the per-task log now. - **`--skip-nx-cache` on the CD lambda build is deliberate and predates the cloud** (`cd-lambda-pulumi.yml`, `adhoc-pulumi-refresh.yml`): a deploy builds its own artifact rather than trusting a cache entry. Dropping it would let CD reuse the exact bundle CI built — a real Nx Cloud win, but it changes artifact provenance for prod, so decide it on purpose. - **Parallel sub-jobs**: `pr.yml` and `main.yml` split the former monolithic Check/Build into three parallel jobs (`lint`, `typecheck`/`build+specs`, `test`) that all `needs: [setup]`. Each pays its own toolchain setup + `bun install`; downstream gates that previously depended on `[build]` now need `[lint, build, test]` so deploys still wait for tests. - **Container build runner**: `ci-container-pulumi.yml` and `cd-container-pulumi.yml` accept a `runner` input, default `medium`, and **no caller overrides it** — `small` is the only other valid value and nothing here wants it. `temporal-worker` and `internal-dashboard` (200s+ TS multi-stage builds) used to pass `runner: large`; measured, they peak near 4.2 GiB with `loadMax` under 5, so `medium` covers them. `admin-dashboard` is not a caller at all — it deploys via SST + OpenNext and has no Dockerfile or `image-build` target. Lambdas use the bundled-zip path (no docker). - **`test` must not `dependsOn: ["^test"]`.** It did until [#3011](https://github.com/CFXLabsInc/cfx-platform/pull/3011), which made every app-test leg (`nx run :test`) transitively run the **entire dependency graph's** test targets — `card-api:test` ran `db:test`, `transaction-services:test`, `services:test` and more. Since each app leg is its own runner and the root `nx.json` sets `parallel: 5`, the container-heavy suites (`createValkeyTestContainer`, `createPostgresTestContainer`) booted repeatedly and concurrently on every leg. The symptom is not a clean OOM but Valkey-backed integration tests failing on eventual-consistency polls that starve — e.g. `TransactionSearchCache` and db's `service-cache` distributed-sync test, both of which pass in isolation in under 2s. **Two different legs failing the identical assertion is the signature.** Dependencies get their own legs from `nx affected -t test`, so `^test` only ever bought redundant execution; `build` / `build-workflows` / `typecheck` stay and still supply artifacts (`build` carries its own `^build`). - **Affected targets**: always `bun nx affected -t `, in every job in both `pr.yml` and `main.yml`. The `tools/scripts/nx-affected.sh` wrapper is gone; there is no `run-many` variant and no project-list plumbing between jobs. - **`nx_affected_override` lives only on `deploy-env.yml`.** To deploy a specific set of projects, dispatch `deploy-env.yml` directly — it takes the comma-delimited list and resolves its own fan-out matrix. `main.yml` used to accept and forward the same input; that was removed because dispatching `main.yml` re-runs release/publish/typedoc too, and in 11 months not one such dispatch ever completed successfully. A `main.yml` dispatch now takes no inputs and runs the whole pipeline against `nx affected`. - **Don't reach for `nx run-many --projects=` as an "affected" substitute.** `--projects=""` reads empty as _no filter_ and runs **every** project — 74 of them, which on the Test job is an instant OOM. `nx show projects --projects=""` returns `[]`, so it does not warn you about the asymmetry. - **Verify before fixing**: never guess role names, secret names, env names, or OIDC subs. Grep actual workflow + IaC files; ask if unverifiable. (See `/preflight` skill.) ## Workflow files **Everything is local.** The 11 reusable workflows and 3 composite actions this repo used to call from `CFXLabsInc/cfx-github-workflows` are vendored into `.github/`; that repo is frozen. Consequences worth knowing: - **Call reusable workflows by relative path** — `uses: ./.github/workflows/cd-pulumi.yml`, with **no `@ref`** (a local ref must not carry one). They run at the calling commit, so a PR that edits a reusable workflow now tests its own version — which the old `@main` pin never did. - **Workflow → composite action refs are relative too** (`./.github/actions/`), for the same reason. No composite action references a sibling action, which is the one case that could not be relative — `./` inside a composite resolves against the caller's workspace, not the action's, so a sibling ref would have to be absolute and `@main`-pinned (and therefore unexercised by the PR editing it). Keep it that way. - **cfx-demo and cfx-marketing still call `cfx-github-workflows`.** They are unaffected by anything here. Repointing them at this repo additionally requires flipping cfx-platform's Actions access from `none` to `organization` (`gh api -X PUT repos/CFXLabsInc/cfx-platform/actions/permissions/access -f access_level=organization`) — without it a cross-repo `uses:` cannot resolve, because this repo is private. - **`ci-sst-nextjs.yml` / `cd-sst-nextjs.yml` keep a GitHub-hosted mode** (`runner_env: ""` → `ubuntu-latest`, bun from the `bun_version` input). That path is the only place `oven-sh/setup-bun` survives, because bun is ambient only on the RunsOn AMI. Don't delete it while off-fleet consumers exist. | File | Trigger | Purpose | | ------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `pr.yml` | `pull_request` | PR gate: format, lint, typecheck, test, gen/validate specs + per-project Pulumi/Vercel previews. | | `main.yml` | `push` to `main` | Build, release (semver), publish specs to Readme + Typedoc to Cloudflare Pages, deploy dev. | | `deploy-env.yml` | `workflow_call` / `_dispatch` | Fans out to lambda/container/vercel/dbmate CD per affected project for one environment. Its infra Pulumi matrix comes from `ci-matrix.ts cd`. | | `cd-lambda-pulumi.yml` | `workflow_call` | Build Lambda image + `pulumi up` for one project. | | `cd-container-pulumi.yml` | `workflow_call` | Build container + `pulumi up` for Fargate workers. | | `cd-pulumi.yml` | `workflow_call` | `pulumi up` only (no app build) for infra-only stacks (s3, db, dashboards). | | `cd-dbmate-aurora.yml` | `workflow_call` | `dbmate up` against Aurora (OIDC + direct VPC connection + IAM auth). | | `ci-lambda-pulumi.yml` | `workflow_call` | PR equivalent of `cd-lambda-pulumi`: build + `pulumi preview`. | | `ci-container-pulumi.yml` | `workflow_call` | PR equivalent of `cd-container-pulumi`. | | `ci-pulumi.yml` | `workflow_call` | PR equivalent of `cd-pulumi`. | | `ci-dbmate.yml` | `workflow_call` | Validates `packages/db/migrations/**` when changed. | | `adhoc-release.yml` | `workflow_dispatch` | Ad-hoc semver bump + release for a single package. | | `adhoc-pulumi-refresh.yml` | `workflow_dispatch` | Ad-hoc `pulumi refresh` for one project + stack, or `nx_project: all` to fan out across every package with a `Pulumi..yaml`. Optional `--clear-pending-creates`. | | `adhoc-clear-runs-on-cache.yml` | `workflow_dispatch` | Purges the RunsOn Magic Cache S3 bucket (`runs-on-fleet-cache-…` in the RunsOn account). Optional `ref` and `key_filter` substring filters, `dry_run: true` by default. **The bucket has no per-repo dimension** — real keys are `cache/v1////` with an empty repo segment, shared with every other repo on the fleet, so you cannot scope a purge to cfx-platform. Auth is the runner's own EC2 instance profile, so it must stay on a RunsOn fleet; no OIDC. For invalidating a poisoned entry, not for cost — `cache_expiration_days` handles expiry. | | `ci-sst-nextjs.yml` | `workflow_call` | PR preview deploy for an SST + OpenNext dashboard (`pr-` stage + URL comment). Local reusable workflow, matrix-driven per app from `pr.yml`'s `sst-preview` job (admin-dashboard, bank-dashboard). Deploy-only — teardown is the reaper. Its CD sibling `cd-sst-nextjs.yml` is called from `deploy-env.yml`. | | `cron-sst-preview-reaper.yml` | `schedule` (12h) / `workflow_dispatch` | Reaps orphaned `pr-` SST preview stages whose PR is no longer open. **Every SST app must be in exactly one of its three lists** — `APPS` (built here, removed with `sst remove`), `LEGACY_APPS` (package gone, direct teardown by tag), `FOREIGN_APPS` (another repo's reaper owns it); an app in none is reaped by nothing and is reported by the `guard` job. It is the primary teardown for the deploy-only `ci-sst-nextjs.yml` previews. Prevents the shared dev-account CloudFront cache-policy quota (default 20) from filling with leaked previews. | ## Common pitfalls - A `cd-*` workflow runs `pulumi up`; a `ci-*` workflow runs `pulumi preview`. Don't cross the wires. - New CD workflows must be wired into `deploy-env.yml`'s fan-out, otherwise they won't run. **A new infra Pulumi _stack_ is the exception — it joins automatically.** Both `pr.yml`'s preview fan-out and `deploy-env.yml`'s infra CD matrix are derived by `infraProjects` in `tools/scripts/ci-matrix.ts` from the presence of `packages/

/pulumi/Pulumi.yaml` plus the per-stack `Pulumi..yaml` files. The CD matrix was a hand-maintained jq literal until then, which is how `cfx-events` previewed on every PR for weeks without ever deploying. - **Three exceptions are policy, not fact, and are named in `ci-matrix.ts`.** `ORDERED_CD_STACKS` (`ecr`, `db`) keep their own hand-written jobs because container CD and dbmate `needs:` them by name; `EXCLUDED_CD_LEGS` holds `(project, stack)` pairs whose stack file exists but must not deploy (today: `temporal-lambda-worker:prod`, committed ahead of its cutover); `EXCLUDED_PREVIEW_PROJECTS` holds projects dropped from the PR preview fan-out entirely (today: `temporal-lambda-worker` — prod is not cut over and dev was a one-off bootstrap, so previewing either stack diffs something nobody will deploy). Deriving purely from file existence would run all of them — if you add a stack file before its cutover, add the exclusion in the same commit. - **`temporal-lambda-worker` is classified as an INFRA stack, not a lambda**, despite the name: it carries no `tag:lambda`, so `infraProjects` claims it. That is why its preview exclusion filters `infra` rather than `previewApp`. Run `nx show projects --projects tag:lambda` before assuming which list a project lands in. - **The `dbmate` job runs against pglite, not a container.** `pr.yml`'s `dbmate` job starts `db:start:bare` (an in-process pglite served on `:54332`, empty — no migrations, no seed) and runs `migrate-up` → `seed` → `migrate-down:ci` against it. Two things make that work and are easy to break: the port must stay `54332` because the committed `packages/db/.env.local` pins `DATABASE_URL` there (the postgres container it replaced published `:5432` and was therefore never reachable — the job failed on `connection refused` every run), and the server must stay **bare**, since applying migrations at boot is exactly what the job is trying to validate. `DBMATE_NO_DUMP_SCHEMA=true` in that env file is also load-bearing: a schema dump would shell out to `pg_dump`, which has no pglite equivalent. - The `dbmate` matrix is built from a hardcoded `["db", "admin-db"]` in `tools/scripts/ci-matrix.ts`; only `db` exists as a project today, so `admin-db` never produces a leg. - **No fleet carries the `ecr-cache` extra any more, and re-adding it would not help.** It rode on `large` and died with it. It did nothing for most of that fleet's life anyway (the module gates the ephemeral registry on `enable_ecr`, unset until 2026-08-11), and it is structurally the wrong tool: the registry it provisions is **ephemeral and per-run**, so it cannot hold cross-run cache, and `cd-container-pulumi.yml` builds and deploys in one job so there is no inter-job handoff either. Cross-run buildkit cache rides on `--cache-from`/`--cache-to=type=registry` against the per-project `cfx--buildcache` ECR repo in the **workload** account — set by `image-build-push` in `nx.json` and threaded via `container-build-push`'s `cacheref` arg. That repo is separate from, and MUTABLE unlike, the `cfx-` artifact repo beside it: the artifact repos are `IMMUTABLE` for Vanta's "Fargate deploys version-controlled images" control, and a cache export rewrites one tag on every build. Lambdas are bundled (esbuild → zip) and need no ECR at all. - Pulumi-only or non-build preview jobs (`ci-pulumi.yml`) use `small` — they only run `pulumi preview`. - **RunsOn runners have no ambient AWS credentials.** Any step calling `aws` must have `id-token: write` on its job plus an `aws-actions/configure-aws-credentials@v6` step. `permissions:` is per-job and not inherited, so splitting a job or adding an AWS step to an existing one silently drops OIDC. A best-effort step that swallows stderr will then "succeed" into its fallback forever. - **All container images — base images AND testcontainers — resolve through the runner's transparent Docker Hub mirror, with one class of exception.** The `docker/Dockerfile.node` builder stage, the `packages/utila-cosigner/Dockerfile` builder and busybox stages, and `packages/db/test/`'s bare `postgres:17` and `valkey/valkey-bundle:9.1` all use bare `FROM node:24-trixie-slim`-style references. The `ecr-pull-through` extra plus the `docker-hub` rule in cfx-self-hosted-services rewrite those to the **fleet** account's ECR and pull over VPC endpoints — same digests, no Docker Hub rate limit, nothing for a consumer repo to configure. Verified by `docker-hub/library/node` and `docker-hub/testcontainers/ryuk` existing in that account's ECR, created by exactly those bare pulls. The **distroless runtime stages are the one class of exception**: `docker/Dockerfile.node`'s `gcr.io/distroless/nodejs24-debian12:nonroot` and `packages/utila-cosigner/Dockerfile`'s `gcr.io/distroless/static-debian12:nonroot` are fully-qualified non-Docker-Hub references, so they bypass the mirror entirely and pull straight from Google's registry over the public internet — no VPC/mirror config applies to them, and none is needed (anonymous GCR pulls aren't rate-limited the way Docker Hub's are). - **The workload-account `ecr-public` machinery is gone**: the pull-through rule, `cfx-role-ecr-pullthrough-ci-`, `tools/scripts/warm-testcontainer-images.sh`, the `CFX_TEST_IMAGE_REGISTRY` env var, `--baseImagePrefix`, and the `id-token: write` those jobs carried only to assume that role. The `ecr-public/*` repositories it populated are deleted too — they outlived the rule by weeks, and Vanta scans every image in an ECR repo, so an abandoned cache reports CVEs against base images no build has read since 2026-07-14. Do not reintroduce a per-repo image prefix — if a pull is rate-limited or slow, the mirror is the thing to fix, in the fleet account. - **A `postgres:17` image-pull 404 is usually NOT an ECR problem.** `open /var/lib/docker/tmp/GetImageBlob: no such file or directory` mid-pull meant RunsOn's `/var/lib/docker` bind-mount had shadowed dockerd's `tmp/` scratch dir. RunsOn now mounts `/var/lib/docker` in its own `runner-setup-mount-points` step before anything else runs, so this should no longer occur; if it does, it is a RunsOn regression, not something to patch around in a workflow. Diagnose from the **raw** step log. - **"Could not find a working container runtime strategy" (testcontainers) and "failed to initialize builder" (buildx) both mean dockerd is DEAD**, not that an image is missing. Check `Cannot connect to the Docker daemon at unix:///var/run/docker.sock` in the raw log. The known cause was two dockerd restarts ≈1s apart — our `preinstall` repair plus RunsOn's `runner-setup-ecr-pull-through-cache` — tripping systemd's start rate limiter (`StartLimitBurst`, 5 starts / 10s). The repair is gone, so anything restarting dockerd during runner setup is suspect. See cfx-self-hosted-services#83. - New runner shapes go in `cfx-self-hosted-services/projects/runs-on-fleet/catalog.tf` (both the `runners` and `fleets` maps) **and** every consumer repo's `.github/actionlint.yaml`'s `self-hosted-runner.labels` list — actionlint is per-repo and cannot see the catalog. There is no org-wide profile file to update: `CFXLabsInc/.github-private` was deleted, and nothing replaced it. - When splitting a previously-monolithic job into parallel sub-jobs, audit every downstream `needs:` in the same workflow — a deploy that used to gate on the monolithic job will silently skip the new sub-jobs unless its `needs:` list is updated to include all of them.