# Secrets and config in SSM Parameter Store

Every value an app reads lives in SSM Parameter Store, one SecureString per key.
A deployed TypeScript app reads the keys it declares from one flat map, baked
into its artifact at deploy and refreshed from its Valkey bundle. Its function's
environment holds only the keys a library reads itself (`libraryEnvironment`). A host script reads SSM on
its sync, CI reads it through the job's AWS role, and a laptop reads it through
[`tools/secrets/run.ts`](../tools/secrets/run.ts).

## Layout

| What                         | Where                                                                                                 |
| ---------------------------- | ----------------------------------------------------------------------------------------------------- |
| A group's operator keys      | `/cfx/secrets/<group>/<KEY>`, one value per account, written by hand or from the admin dashboard      |
| A group's stack keys         | `/cfx/stack/<stage>/<group>/<KEY>`, written by the deploy that creates the resource                   |
| What a stack reads at deploy | `/cfx/secrets/<project>/<KEY>` (`common`, `hookdeck`, `wallet-api`; below)                            |
| A branch config              | one level deeper: `/cfx/secrets/bank-dashboard/dart/<KEY>`, `/cfx/secrets/movemoney-app/dashfi/<KEY>` |
| Dev values                   | the workload-dev account (AWS profile `workload-dev`)                                                 |
| Prod values                  | the workload-prod account (`workload-prod`)                                                           |

The other shared projects are `ci` (what GitHub Actions needs before it can
reach anything else) and `solana-keypairs`, which only the wallet-api backfill
script reads and which awaits deletion.

Each value is a SecureString under `alias/aws/ssm`, tier `Intelligent-Tiering`,
stored without a trailing newline. SSM refuses an empty value, so a key with no
value has no parameter. Reads are non-recursive: a branch config's keys are not
part of its parent's.

`/cfx/secrets/` is owned by no stack. It is written by hand, from the admin
dashboard's `/config`, and by
[`AppSecrets.putSecret`](../packages/pulumi-templates/src/AppSecrets.ts) for a
credential a program mints that another stack reads by name at deploy (the
Cognito staff clients, the Outpost key).
`/cfx/stack/` is written only by deploys.

## Adding or rotating a value

Find the key's group and owner first:

```bash
rg -n '<KEY>' packages -g configNamespace.ts
cfx dev config show <group>
```

An operator key is written at its group's path:

```bash
aws ssm put-parameter --profile workload-dev --region us-west-2 \
  --type SecureString --tier Intelligent-Tiering --overwrite \
  --name /cfx/secrets/<group>/<KEY> --value file:///path/to/value
```

`file://` keeps the value out of shell history; the file holds no trailing
newline (`printf %s`, not `echo`). Run it again with `--profile workload-prod`
for prod. Every service whose key list holds it sees the new value within about
30 seconds (below). The admin dashboard's `/config` does the same write with an
audit record.

A stack key is never written by hand: change the `writeStackConfig` call in the
stack that owns it, and deploy that stack.

## How an app declares its config

**A key is declared by the code that reads it.** A services package declares
its keys in `src/configNamespace.ts` with `defineConfigKeys`
([`service-config`](../packages/service-config/README.md)), and a key only one
app reads goes in that app's own (`customer-dashboard`, `temporal-worker`,
`mapbox` in deposit-api):

```ts
// packages/temporal/src/configNamespace.ts, abridged
export const temporalConfig = defineConfigKeys({
  group: "temporal",
  keys: {
    B2B_TEMPORAL_WORKER_TASK_QUEUE: {
      owner: "operator",
      plain: true,
      default: "b2b-temporal-worker",
    },
    TEMPORAL_FUNCTION_NAME: { owner: "stack", plain: true },
  },
  writtenBy: ["cfx-temporal-api", "cfx-temporal-cloud", "cfx-temporal-worker"],
});

const fn = temporalConfig.get("TEMPORAL_FUNCTION_NAME"); // synchronous
```

- **A key keeps its full environment-style name**, unique across every
  declaration (`test/declarations.test.ts`), so the SSM path, `local.env` and
  `rg` all spell it the same way. A process reads one flat map of them.
- **`group`** names only the SSM path a key lives under. Runtime code never
  names a group.
- **`owner`** picks the tree: `operator` (`/cfx/secrets/<group>/`) or `stack`
  (`/cfx/stack/<stage>/<group>/`, written by a `writtenBy` stack).
- **`required`** defaults to `true`. `required: false` only where the code has a
  working path for the key's absence (`TEMPORAL_API_KEY`: unset is a local dev
  server), or a stage lacks it on purpose (Dart's SFTP outside prod).
- **`default`** applies when no tree holds the key. A `""` default is refused.
- **`plain`** marks a value safe to display; every other key is masked in the
  CLI, the admin page and errors. In SSM every value stays a SecureString.
- **Every value is a string.** Convert where it is used.
- **`get(key)` is synchronous and throws** with the key and, when deployed, the
  paths to set; `getOptional(key)` returns `undefined`. Never `await` either.

After adding or changing a declaration, run `bun nx run service-config:gen-manifest`:
it regenerates `manifest.json`, `src/names.ts` and `readers.json`, and a test
fails when they drift.

**A workload lists the groups it reads** in
[`workloadConfig.ts`](../packages/pulumi-templates/src/workloadConfig.ts)
(`DEPOSIT_API_CONFIG`, …, collected in `WORKLOAD_CONFIGS`, keyed by service).
Every key of those groups is on the service's key list: its deploy resolves and
bakes them, its bundle holds them, `tools/secrets/run.ts` reads their trees
locally, and `readers.json` names the service as a reader. A key off the list
is absent at run time, and `get()` throws naming `workloadConfig.ts`.

### Deploy keys

A **deploy key** is declared `owner: "deploy"`: it has no SSM path, and each
function's deploy bakes its own value into `config.json`. They are the
Postgres and Valkey logins (`database`, `valkey` groups), `CONFIG_STAGE`,
`CONFIG_SERVICE`, `APP_MODE`, `SERVER_PORT`, `LOG_LEVEL`, `ENVIRONMENT`,
`MAINTENANCE_SERVICE_ID` and `CFX_APP_NAME` (`process` group). Everything else
is an ordinary declared key.

- **Code reads one like any key**, synchronously:
  `processConfig.getOptional("SERVER_PORT") ?? "3000"`, keeping a local
  default at the call site.
- **The stack bakes it.** `declareLambdaArtifact(name, { path, config, deploy,
functions })` writes `config.json` as `{ values, deploy, functions }`:
  `deploy` is `deployConfig({ ...configReaderEnv({ stage, service }), ... })`
  for every function of the artifact, and `functions` holds a function's own
  logins by its Lambda name. `deployConfig` throws on a key that is not
  deploy-owned.
- **The environment holds library keys only.** `libraryEnvironment` admits
  `HOME`, `SENTRY_DSN`, `SENTRY_ENVIRONMENT` and the `TEMPORAL_*` connection,
  which a library reads from `process.env` itself. Lambda's own variables
  (`AWS_REGION`, `AWS_LAMBDA_FUNCTION_NAME`) are read with `lambdaEnv(...)`.
- **The code key ARN** is read from SSM `/cfx/kms/lambda-code-key/arn` (the s3
  stack writes it); a preview before it exists plans without KMS.
- **Nothing else reads `process.env`.** `n/no-process-env` refuses it outside
  `tools/lint/process-env-allowlist.txt`; `process.env.NODE_ENV` is allowed.

## Runtime config: baked at deploy, refreshed from Valkey

**At deploy**, `deployServiceConfig({ service, stage, dependsOn })`
([`stackConfig.ts`](../packages/pulumi-templates/src/stackConfig.ts)) runs in
every stack that sets `CONFIG_SERVICE`, after the stack keys it writes:

- **It resolves every key on the service's list** with `resolveConfig`, the
  function the refresher uses: the stack value, else the operator value, else
  the default. It fails the apply naming every required key with no value; a
  preview warns instead.
- **It bakes the result.** `declareLambdaArtifact(name, { path, config })` writes
  it beside `bootstrap` as `config.json`, uploads the zip to
  `cfx-lambda-artifacts-<stage>` under `cfx-lambda-code-key`
  (`alias/cfx-lambda-code`, in the `s3` stack), and puts that key on the
  function as `sourceKmsKeyArn` and `kmsKeyArn`. Only `cfx-role-devops`, the
  `cfx-role-*-exec` roles and Lambda itself can decrypt it, so `GetFunction`
  returns no code to anyone else.
- **It writes the service's key list** to
  `/cfx/stack/<stage>/consumers/<service>` and runs the gate (below).

**The three dashboards bake too.** Their `sst.config.ts` resolves the same keys
with `writeSstBakedConfig` before the site is built and adds `config.json` to
the server zip through `copyFiles`, never to the cached `open-next-build`
output. SST uploads that zip to its own `sst-asset-*` bucket, whose default
encryption the `s3` stack sets to the code key; the server function carries the
key like a Lambda. The Temporal workers bake like every other Lambda: Temporal Cloud holds
`lambda:GetFunction` on them, and gets an error, not the code, because it
cannot decrypt the code key. Besides the deploy, execution and dashboard server
roles, the key admits the SSO sets that decrypt SSM secrets in that account
(`SSO_SECRET_READER_SETS` in `packages/s3/pulumi/lambdaCodeKeyPolicy.ts`), so an
operator who can read a secret in SSM can read it baked. A `pr-<N>` preview bakes dev's values with its own
`deploy` keys over them (its database, keyspace and pool sizes), carries no
`CONFIG_SERVICE`/`CONFIG_STAGE` so it never refreshes, and takes no code key.

**At run time** (`ServiceConfigService`, `@cfxlabsinc/service-config/service`):

- **Deployed** (`CONFIG_SERVICE` set on Lambda), a process serves
  `$LAMBDA_TASK_ROOT/config.json` from its first read. **Locally** (and in
  `next build`), every key resolves from `process.env`, then its default.
- **A snapshot older than 30 seconds fires one background refresh** of
  `cfg:svc:{<service>:<stage>}`: the `ver` key, and the bundle only when it
  moved. Reads never wait on it. There is no timer.
- **A refreshed bundle that does not parse, or drops a required key, is
  refused**: the process keeps serving what it has and logs
  `Refused config bundle version <n>: <reason>` once. Valkey unreachable is the
  same: it keeps serving. The required keys are the `required` list the deploy
  baked into `config.json` (required with no default, from the manifest), not
  whatever namespaces the process has loaded, so an optional key drops freely.
- **The process holds no SSM grant.** Its Valkey login reads its own bundle
  only (`%R~cfg:svc:{<service>:*`, [CACHING.md](CACHING.md)); the admin
  dashboard's reads every bundle for `/config`.

**The two trees and their owners.**

- **Only `cfx-role-devops` writes `/cfx/stack/`.** The SCP
  `cfx_scp_deny_stack_config_writes`
  ([`aws-organisation`](../packages/aws-organisation/service-control-policies.tf))
  denies `PutParameter`, `DeleteParameter(s)`, `LabelParameterVersion`,
  `UnlabelParameterVersion`, `AddTagsToResource` and `RemoveTagsFromResource`
  there to every other principal in both workload accounts, SSO roles
  included. A laptop `pulumi up` of a stack that writes stack keys fails on
  them; deploy it from CI.
- **`writeStackConfig`** ([`stackConfig.ts`](../packages/pulumi-templates/src/stackConfig.ts))
  writes a stack key in the same apply that creates its resource, and refuses a
  key its group does not declare `owner: "stack"`.
- **A value resolves stack first, then operator, then default.** A key set in
  both trees takes the stack's.

**The refresher.** Any write under either tree fires an SSM `Parameter Store
Change` event, and `config-refresher` rebuilds within seconds every service
whose key list holds the key; a write to a key list rebuilds that service. An
hourly run rebuilds every service, which covers a lost event. A rebuild that
would leave a required key with no value is refused, keeping the last good
bundle, and alarms on the ops topic naming keys, never values
([runbooks/config-bundle-refused.md](runbooks/config-bundle-refused.md)).

**The deploy gate.** After writing the key list, the deploy invokes the
refresher for its service and fails with
`Config gate refused <service>/<stage>: <reason> (<keys>)` when the bundle is
refused, so `live` never shifts onto an incomplete one. It sends the list's
`keysHash`, and the refresher refuses one that differs from the manifest it was
built with, so `config-refresher` deploys first: every gated stack lists it in
`deploy.after`, and it deploys after `s3`, which owns the code key. A throttle
or timeout is retried three times, then fails by name. A dashboard's gate is in
its Pulumi stack, which applies before `sst deploy`.

**Order between stacks.** A deploy reads keys other stacks write, and
`deploy.yml` applies the selected stacks in one parallel call. A stage's first
deploy of an app needs the writer stacks (each group's `writtenBy`) deployed to
it first; otherwise its resolve fails on the missing key, and a re-run passes.

**Before a merge.** Two checks catch a key a deploy would otherwise refuse only
after the PR has landed:

- **A stack-owned key needs a writer.** `tools/ci/stack-config-writers.test.ts`,
  in `Checks`, fails on a required key with `owner: "stack"` and no default that
  no `writeStackConfig` call writes, naming the key and its group. A call
  whose group or keys are not literals fails too, because the test cannot
  read it.
- **An operator-owned key needs a value.** The `Config presence` job in
  `pr.yml` fails when a required operator key with no default has no value in
  dev or prod, for the whole manifest on every PR. It lists names with
  `ssm:DescribeParameters`, which returns no values, and prints the key, stage,
  path and command. Set the value, then re-run the PR's failed jobs:

  ```bash
  AWS_PROFILE=workload-prod aws ssm put-parameter --type SecureString --tier Intelligent-Tiering --name /cfx/secrets/<group>/<KEY> --value '<value>'
  ```

  Where a stage genuinely has no value, correct the declaration instead:
  `required: false`, a default, or a deploy key. A fork PR gets no AWS
  credentials, so this job fails on one; open the PR from a branch in this
  repository.

**Previews.** `pr-closed.yml` runs
`packages/config-refresher/scripts/config-teardown.ts pr-<N>`, which deletes the
stage's `/cfx/stack/pr-<N>/` keys and any `cfg:svc:{*:pr-<N>}:*` bundles.

**Who can read a baked secret.** `s3:GetObject` on the artifact bucket is not
enough: the object is under the code key. The readers are the principals that
may decrypt with it (`cfx-role-devops`, every `cfx-role-*-exec`, Lambda) and,
for a bundle, its service's own Valkey login, the admin dashboard's and
`cfx-ops`. `lambda:GetFunction` is held only by the deploy role and the three
roles Temporal Cloud assumes to invoke a worker
(`tools/ci/lambda-code-readers.test.ts`); on a baked function it returns no
code to them.

### Seeing and editing config

- **`cfx <env> config show <group|service>`** lists each key's owner, whether it
  is required and set, its SSM version and last change, and the value of a
  `plain` key; a secret prints `••••`. A service shows its bundle, the keys its
  last deploy listed, and any declared key that deploy did not list.
  `--reveal KEY` prints one decrypted value and is kept out of the CLI's
  history. It reads with your profile and never decrypts a listing.
- **The admin dashboard's `/config`** (`config:r`) shows the same per stage,
  the services that read each group from
  [`readers.json`](../packages/service-config/readers.json), and each key's
  last 100 versions with who wrote them; values only for `plain` keys.
- **Saving on `/config`** (`config:w`) writes an operator key as a new version
  under `/cfx/secrets/<group>/<KEY>` and records `config.updated` in the
  admin activity log, with both values for a `plain` key and only SHA-256
  prefixes for a secret. Revert writes an earlier version's value as a new one.
  Stack keys are read-only there, and the SCP denies the dashboard role
  `/cfx/stack/` writes anyway. A save or revert needs the operator's passkey,
  confirmed in this session in the last five minutes: without one the dialog
  asks for it, then saves. A preview host can confirm and localhost cannot, so
  locally edit with `aws ssm put-parameter` as above
  ([AUTH.md § Step-up](AUTH.md#step-up)).

## When a value goes live

A `put-parameter` changes the parameter; what happens next depends on what the
value is and who reads it:

| Value                                    | Read by                                                                       | A change lands                                                    |
| ---------------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------- |
| A declared key, in a Lambda or dashboard | `ServiceConfigService`: baked, then its bundle                                | within seconds of the refresher's rebuild, plus the 30-second TTL |
| A deploy key                             | the function's `config.json`, baked by its stack                              | on that stack's next deploy                                       |
| A `wallet-api` key                       | the function's environment, inlined from `/cfx/secrets/wallet-api/` at deploy | on that stack's next deploy                                       |
| `utila-cosigner` (EC2 host)              | the host's own scripts (`vault-sync.sh`), from SSM, on each sync              | on the next sync                                                  |
| A laptop                                 | `tools/secrets/run.ts` at launch (`local` adds `local.env`)                   | on the next launch                                                |

A declared key needs no deploy and no cold start: a running process picks it up
on its first read after its snapshot turns 30 seconds old, and the next deploy
bakes it. A deploy key is the opposite: nothing re-reads a function's
`config.json`, so it changes only when the stack that bakes it deploys. See
[runbooks/secret-value-not-live.md](runbooks/secret-value-not-live.md) for each
case.

**A dashboard** reads declared keys like a Lambda, from the bundle it loads at
startup. SST gives one `environment` to both `next build` and the server
function, so the build carries only `NEXT_PUBLIC_*` and the keys Next reads, but `next build` runs on no Lambda, reads keys from
`process.env`, and needs no AWS or Valkey to pass. The deploy-time check is the
gate in the dashboard's Pulumi stack; a `pr-<N>` preview has none. At deploy
its `sst.config.ts` reads, with the deploy role:

- `NEXT_PUBLIC_*` from `/cfx/secrets/<dashboard>/` (bank and customer), copied
  into the build's `process.env` and inlined into the client bundle. A change
  to one needs a redeploy of the dashboard.
- Its declared keys, resolved as a Lambda's are and baked into the server's
  `config.json` with its deploy keys (`deploy`). A `pr-<N>` preview bakes
  dev's values, with its own overrides in `deploy`.
- `COGNITO_PLATFORM_USER_POOL_ID` / `COGNITO_BANK_USER_POOL_ID` from
  `/cfx/secrets/cognito/`, only to scope its IAM statements.
- The Postgres and Valkey logins from `/cfx/db/…` and `/cfx/valkey/…`.

A local `next build` takes every declared key from `process.env` (`local.env`,
or `local.offline.env` with `-c local-offline`).

**Rust Lambdas** (`wallet-api`, `pdf-renderer`) read their plain environment
through `Config::from_env` (`src/config.rs`). `wallet-api` (ENG-3883) still has
its values inlined at deploy from `/cfx/secrets/wallet-api/`: its stack reads
the path with [`AppSecrets`](../packages/pulumi-templates/src/AppSecrets.ts) and
inlines the stored values of the keys its own list names (`SOLANA_RPC_ENDPOINT`,
`EVM_RPC_ENDPOINTS`, …) beside its stack values, so a change there needs a
redeploy. `pdf-renderer` has no operator values: its stack sets every key.

**EC2 host.** `utila-cosigner` runs as systemd units on one host per account
([`host.ts`](../packages/utila-cosigner/pulumi/host.ts)). Its scripts in
`packages/utila-cosigner/host/` read the parameters they need (the database
URL, for one) from SSM on each `cfx-vault-sync` run, so a changed value lands on
the next sync with no deploy.

## Deploy credentials

A stack reads the credentials its providers need with
[`requireSecret(key, { project? })`](../packages/pulumi-templates/src/commonSecret.ts),
which reads `/cfx/secrets/<project>/<KEY>` (default `common`) in the stack's own
account and fails the deploy when it is empty. `sentryAuthToken()` is that for
`SENTRY_AUTH_TOKEN`. The Cloudflare token is the exception: it is one parameter,
`/cfx/ci/cloudflare-api-token`, per account (see
[runbooks/cloudflare-api-token-rotation.md](runbooks/cloudflare-api-token-rotation.md)).
A deploy credential reaches no function.

## CI

CI authenticates to AWS with GitHub OIDC as the workload account's
`cfx-role-devops`, which holds `ssm:*` on every parameter in its account (AFT
`workloads-*/terraform/iam.tf`), so any job that assumes it, PR jobs included,
can read every secret. There are no other secret identities.

- [`.github/actions/ssm-secrets`](../.github/actions/ssm-secrets/action.yml)
  takes a `path` and a list of `keys`, reads only those, masks them and exports
  them to `$GITHUB_ENV`. A configure-aws-credentials step must already have run.
- `warp-connect` reads the WARP service token from `/cfx/secrets/ci/`.
- `/cfx/secrets/ci/` holds what a job needs before it can reach anything else:
  the WARP service token, the Docker Hub pull credentials, `SENTRY_AUTH_TOKEN`,
  `LINEAR_API_KEY`, `FIREWORKS_API_KEY` and the e2e users' credentials.
  `common` holds the same deploy credentials a stack reads, so rotating one of
  the three duplicated values means writing both paths.

See [TEMPORAL_CLOUD.md](TEMPORAL_CLOUD.md) for the Temporal credential a stack mints.

## Local development

Locally `CONFIG_SERVICE` is unset, so every declared key is read from
`process.env` by its full name. [`packages/nx-secrets`](../packages/nx-secrets/README.md)
infers `secrets-run`, `secrets-serve`, `secrets-build` and `secrets-serve-lambda`,
which fill that environment through `tools/secrets/run.ts`.

| Configuration     | Source                                                                                  |
| ----------------- | --------------------------------------------------------------------------------------- |
| `local` (default) | dev's SSM parameters, then the package's `local.env`, then the ambient environment      |
| `local-offline`   | the package's committed `local.offline.env`, then `local.env`, then the ambient; no AWS |
| `dev`, `prod`     | `bun tools/secrets/run.ts --app <slug> --config <dev\|prod> -- <cmd…>`                  |

```bash
bun nx secrets-serve deposit-api              # dev parameters + local.env
bun nx secrets-serve deposit-api -c local-offline  # dummies + local.env, no AWS
bun nx secrets-serve deposit-api -c dev       # dev parameters only
bun tools/secrets/run.ts --app temporal-worker --config prod -- <cmd…>
```

**What SSM layer a run reads.** For each group in the app's config list
(`WORKLOAD_CONFIGS`), `/cfx/secrets/<group>/` and then
`/cfx/stack/<stage>/<group>/` (`dev` for `local` and `dev`, `prod` for `prod`),
then `/cfx/secrets/<app>/`; a later path wins, and an empty path is no error.
`<app>` is the package name, except where its `project.json` sets `secretsApp`
(`admin-api` reads `admin-dashboard`'s, a services package `temporal-worker`'s).
A slug with no config list (`ci`, `cfx-link`) reads only its own path.

A `local` run reads the same paths as `dev`, so it needs
`aws sso login --sso-session cfx`. Later layers win, and any key whose final
value is the empty string is removed: `KEY=` in a `local.env` unsets a dev key.
SSM holds no deploy key a deploy derives, so a local run takes the Postgres
login (`DATABASE_URL`, the container's; see [DB_CREDENTIALS.md](DB_CREDENTIALS.md)),
`VALKEY_URL`, `TEMPORAL_ADDRESS` (`localhost:7233` for a dev server) and
`TEMPORAL_API_ENDPOINT` (`temporal-api`'s `local-invoke`) from `local.env`. An
app that renders PDFs reaches the renderer through `PDF_RENDERER_ENDPOINT`;
serve it as [packages/pdf-renderer/README.md](../packages/pdf-renderer/README.md)
shows. The root [`.env.example`](../.env.example) has a local value for the
shared ones. A required key with no value throws on its first read, naming it.

The read always uses the config's own profile, `workload-dev` for `local` and
`dev` and `workload-prod` for `prod`, whatever `AWS_PROFILE` the shell exports.
It also refuses a parameter from the wrong account, so a misconfigured profile
cannot hand prod secrets to a local run. The command itself still inherits the
shell's `AWS_PROFILE`.

`local.env` is committed and holds only non-secret local overrides (localhost
URLs, ports, flags). It must never hold a secret;
[`tools/ci/env-local.test.ts`](../tools/ci/env-local.test.ts) fails on a
secret-named key with a value (`DB_PASSWORD`, the local container's, excepted) and
on a URL with credentials for a non-local host, and on a token-shaped value under
any key.

A `local-offline` run reads no SSM and needs no AWS session. In place of dev's
parameters it reads the package's committed `local.offline.env` of dummies,
then layers `local.env` and the ambient environment the same way. A vendor call
then fails on the network, not at boot. A package with no `local.offline.env`
fails with that path. `env-local.test.ts` scans these files too, and a
secret-named value there must start `offline-dummy-`.
[`tools/ci/local-offline.test.ts`](../tools/ci/local-offline.test.ts) fails on a
key in a `local.offline.env` that is neither a deploy key nor a key of a
group the app reads, on a file that belongs to no workload, and on a
workload with no file outside its `WITHOUT_OFFLINE_ENV` list.

The file is deliberately not named `.env.local`. nx loads a project's
`.env.local` into every task, and Next and bun load that name themselves, so a
committed one would reach `-c dev`, `-c prod`, tests and builds wherever SSM or
the deploy lacks a key it sets. Only `tools/secrets/run.ts` reads `local.env`,
and only for `-c local` and `-c local-offline`; the same test fails on a tracked
`packages/*/.env.local` other than the db project's, which its tasks rely on nx
loading.

## Gotchas

- **A key listed under a branch path is invisible to its parent's read**, and the
  reverse. Reads are non-recursive on purpose.
- **A trailing newline is part of the value.** Write from a file made with
  `printf %s`, and compare lengths when a credential "looks right" and fails.
- **An empty value cannot be stored.** To remove a key, delete the parameter
  (`aws ssm delete-parameter`); the bundle then falls to the key's default, or
  the refresher refuses it when the key is required.
- **A parameter in the wrong account is an absent parameter.** `workload-dev`
  and `workload-prod` hold different values at the same path.
- **A value under `/cfx/secrets/<app>/` reaches no deployed TypeScript app.**
  Only a laptop run, `wallet-api`'s stack, a dashboard's `NEXT_PUBLIC_*` copy
  and a credential a stack reads by name read an app's own
  path. Declare the key in the package that reads it.
- **The OIDC role limits CI to `refs/heads/main` or an environment.** A branch
  dispatch fails at the AWS step, not at the read.
- **Never delete a parameter because no code names it.** Two readers build names
  at run time: the wallet-api backfill script in
  [`packages/wallet-api/scripts/`](../packages/wallet-api/scripts/) (`${type}_KEYPAIR`
  under `/cfx/secrets/solana-keypairs/`) and the Hookdeck secrets map in
  [`packages/backoffice-api/pulumi/hookdeck/secrets.ts`](../packages/backoffice-api/pulumi/hookdeck/secrets.ts).

## History

The values were seeded from Doppler and verified by hash. On 2026-10-13 every
Doppler project was snapshotted to the 1Password `Doppler Archive` vault
(`Doppler snapshot 2026-10-13 — <project>`, one `{config: {KEY: value}}`
document each) and then deleted. Retired keys live there, deliberately apart
from the working credentials. The plan and decisions are in
[proposals/DOPPLER_TO_SSM.md](proposals/DOPPLER_TO_SSM.md). The move from
per-app key lists to namespaces, and from there to one baked map per service, is in
[proposals/SERVICE_CONFIG.md](proposals/SERVICE_CONFIG.md).
