# A bank document is in S3 but not in `bank.bank_document` **Last executed in prod 2026-10-07** (the initial backfill): dart 556 files and 13 folders, fib 197 files and 7 folders. Dev the same day: dart 21 files and 3 folders. `bank.bank_document` indexes the files under `s3://cfx-bank-documents-/bank-documents/`: one row per file or folder. S3 holds the bytes, the table holds the listing. Both dashboards list a bank's documents from the table, so a file with no row is in S3 and on no page. `s3-event-handler` also writes rows: S3 tells it about every object created or deleted in the bucket, and within seconds it inserts the row for a new file (and the folder rows above it) or soft-deletes the row of a file that is gone. Every file written or deleted is also published as `bank.document.created` / `bank.document.deleted` on the internal event bus, with data `{ bankId, key }`, by the admin and bank dashboards, `temporal-worker` and the handler alike. The backfill publishes none. This runbook is for the cases the handler leaves, or fails on. ## Symptoms → cause | What you see | Cause | | --------------------------------------------------------- | ----------------------------------------------------------------- | | The table is empty after its migration deployed | The initial backfill has not run | | A file written before the index deployed has no row | Same: the handler only sees objects created after its rule exists | | A Drive file has no row minutes after it reached S3 | The handler failed on it; look in the DLQ | | `cfx-s3-event-handler-dlq-age-` fired | An event failed three times, or EventBridge could not deliver it | | A dashboard upload the browser never completed has no row | The handler leaves signed uploads to `completeUpload`; run Step 2 | | A file copied within S3 has no row | The copy keeps its source's `uploaded-by` mark; run Step 2 | | A file written by an app still on the old task or Lambda | That app had not picked up the deploy; see below | | A report workflow failed after its file reached S3 | The S3 put succeeded and the index write threw | All are repaired the same way. The run reads every object version under each bank's prefix and inserts the rows that are missing. It never updates or deletes a row and never writes to S3, so it is safe to run again at any time. The handler never indexes an object that carries `uploaded-by` metadata, because that is a dashboard user's signed upload and its row belongs to `completeUpload`; a row written first would make `completeUpload` refuse the user's upload. Two consequences. An upload the browser never completes has no row until the backfill runs, which does index marked objects (the next paragraph says how it attributes them). And an object copied within S3 keeps its source's `uploaded-by` mark, for example the copy a bisync conflict rename makes or a restored version, so it also waits for the backfill. The Drive bisync runs without `--track-renames`, so a move or rename in Drive arrives as a delete plus a fresh upload: the old row is soft-deleted and a new one inserted. `created_at` comes from the object's first S3 version and `updated_at` from its latest, so a backfilled row carries the file's real dates, not the time of the run. Every object the run would insert is HEADed for its `x-amz-meta-uploaded-by` metadata, which a signed upload carries. A file a bank user uploaded but never completed gets `updated_by_user_id` set to that user, a member of the bank's workspace. One marked with someone who is not a member of the bank is not indexed and counts in `skipped`; it stays in S3 for you to inspect. An unmarked object, written by CFX staff, a workflow or by hand, gets null, which reads as "CFX". ## Before you start - Cloudflare WARP on ([docs/WARP.md](../WARP.md)); the database is not reachable without it. - An admin SSO profile. The run lists object versions, which needs `s3:ListBucketVersions`, and none of the three apps' roles is granted that on this bucket. - `rds-global-bundle.pem` from AWS for `NODE_EXTRA_CA_CERTS`. - Valkey, over the same WARP tunnel. The listings are cached for 24 hours in the `bank-document-search` namespace ([CACHING.md](../CACHING.md) §9), and a run that inserts clears it. The `VALKEY_*` values below are the ones the post-deploy flush uses (the `Flush non-auth Valkey caches` step in [deploy.yml](../../.github/workflows/deploy.yml)), with the admin tier's user `cfx-admin-` in place of the CD one. The script refuses to start without them, `--check` included. ## Step 1: see what is missing `--check` reads S3 and the table and writes nothing. Dev: ```bash aws sso login --sso-session cfx NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-dev \ env DB_HOST=cfx-db-dev.cluster-cm87kdk8opyp.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ BANK_DOCUMENTS_S3_BUCKET=cfx-bank-documents-dev \ VALKEY_HOST=master.cfx-cache-dev.mueox9.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-dev VALKEY_USERNAME=cfx-admin-dev \ bun packages/bank-services/scripts/backfillBankDocuments.ts --check ``` Prod: ```bash aws sso login --sso-session cfx NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-prod \ env DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ BANK_DOCUMENTS_S3_BUCKET=cfx-bank-documents-prod \ VALKEY_HOST=master.cfx-cache-prod.h4koef.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-prod VALKEY_USERNAME=cfx-admin-prod \ bun packages/bank-services/scripts/backfillBankDocuments.ts --check ``` Expected shape, one line per bank (the counts here are made up), and exit code 1 while anything is pending: ``` dart: 412 file(s) in S3, would insert 412, skipped 0; 6 folder(s), would insert 6. fib: 96 file(s) in S3, would insert 96, skipped 0; 1 folder(s), would insert 1. burling: 0 file(s) in S3, would insert 0, skipped 0; 0 folder(s), would insert 0. ``` `skipped` counts keys whose path the index cannot hold, such as an empty segment (`a//b.csv`), and objects whose uploader is not a member of the bank. They stay in S3 and out of the listing. Add `--bank dart` to run one bank. ## Step 2: insert the missing rows The same command without `--check`. Prod: ```bash NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-prod \ env DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ BANK_DOCUMENTS_S3_BUCKET=cfx-bank-documents-prod \ VALKEY_HOST=master.cfx-cache-prod.h4koef.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-prod VALKEY_USERNAME=cfx-admin-prod \ bun packages/bank-services/scripts/backfillBankDocuments.ts ``` The lines now read `inserted` with the same counts Step 1 reported, followed by one more (the count is made up): ``` Cleared 14 cached bank document key(s). ``` That line is the cache bust. For a bank whose listings were cached and which gained rows, the run also prints `[valkey-cache] invalidateTag failed` with `no permissions to run the 'hdel' command`. That is expected and harmless here: the service busts by tag, and the admin tier's Valkey user may `DEL` but not `HDEL` (`cacheBustAccessString` in [packages/aurora/pulumi/index.ts](../../packages/aurora/pulumi/index.ts)). The script therefore clears the whole namespace itself, which that user is allowed to do. The first bank to gain rows may instead log `Standalone connect failed ... timed out`: the client's first dial over WARP timed out, and it had reconnected by the namespace clear. If the `Cleared` line is missing, the run failed before it: fix what it reports and run Step 2 again. ## Step 3: verify Run Step 1 again. Every line reads `would insert 0` and the exit code is 0. The files now show on the bank's Documents page and under Banks in the admin dashboard. ## A deploy that has not reached every writer Three apps write documents: `temporal-worker` (the report and positive pay workflows), `admin-dashboard` and, later, `bank-dashboard`. A warm Lambda that started before the deploy keeps writing to S3 without a row until it is replaced. Confirm the artifact, not the merge: the `cfx-temporal-worker` function's and the dashboard Lambda's `LastModified`. Then run Steps 1 to 3 once more to pick up whatever the old code wrote in between. ## The handler's dead-letter queue A failed invocation is retried twice by Lambda, then parked in `cfx-s3-event-handler-dlq-`; `cfx-s3-event-handler-dlq-age-` fires while it holds a message. The shared procedure for reading and replaying a parked message is [sns-consumer-dlq-parked.md](sns-consumer-dlq-parked.md) § EventBridge consumers. For this handler, resolve the queue URL once. Prod: ```bash Q=$(AWS_PROFILE=workload-prod AWS_REGION=us-west-2 aws sqs get-queue-url \ --queue-name cfx-s3-event-handler-dlq-prod --query QueueUrl --output text) ``` Dev: ```bash Q=$(AWS_PROFILE=workload-dev AWS_REGION=us-west-2 aws sqs get-queue-url \ --queue-name cfx-s3-event-handler-dlq-dev --query QueueUrl --output text) ``` 1. Read the parked messages without consuming them (use the profile that matches `$Q`): ```bash AWS_PROFILE=workload-prod AWS_REGION=us-west-2 aws sqs receive-message \ --queue-url "$Q" --max-number-of-messages 10 --visibility-timeout 0 ``` 2. Find the error in `responsePayload.errorMessage`, and the bucket and key in `requestPayload.detail`. Fix the cause (a deploy, a grant) before replaying. 3. Replay `requestPayload` with `aws lambda invoke --invocation-type Event`, following [sns-consumer-dlq-parked.md](sns-consumer-dlq-parked.md) § EventBridge consumers. Replaying is the repair and is always safe: a key already in step is `UNCHANGED`. Steps 1 to 3 above stand in only for a parked `Object Created` (`detail-type`, with the bucket in `detail.bucket.name`), and only for the row: they insert missing rows and never update or delete one. A parked `Object Deleted` is not repaired by them (the live row stays and the file opens as not found): replay it, or delete the document from the admin dashboard's Banks page. The handler logs one JSON line per event to `/aws/lambda/cfx-s3-event-handler-`: `{"message":"s3 object event","id":…,"bucket":…,"keySha256":…,"outcome":…,"reason":…}`. The key is never logged, because keys hold file names; `keySha256` is the lowercase hex SHA-256 of the key. To find a file's line: ```bash printf %s '' | shasum -a 256 ``` Then search the log group for that hash. `outcome` is `INSERTED`, `DELETED`, `UNCHANGED` or `SKIPPED`; a skip carries its `reason`: `NOT_A_DOCUMENT_KEY`, `OWNER_NOT_FOUND` or `SIGNED_UPLOAD`. If the Lambda's `Throttles` alarm fires, more than 20 events arrived at once, for example a large folder deleted in Drive. Lambda holds and retries throttled events, so nothing is lost; act only if the DLQ alarm follows. ## Limitation: a lost event The handler writes the row, then publishes `bank.document.created` or `bank.document.deleted` on the internal event bus. If publishing fails after the row is written, Lambda retries the invocation, finds the row already in step and does not publish again. That one event is lost; the row is correct. ## Rollback There is nothing to roll back in S3; the run does not write there. It only adds index rows that mirror objects which exist. If an object should not be there, delete the document from the admin dashboard's Banks page, which removes the object and soft-deletes its row. ## Prevention - [`BankDocumentService.create`](../../packages/bank-services/src/BankDocumentService.ts) writes the row in the same call as the S3 put, S3 first, so a failure leaves an unindexed object this run repairs and never a row pointing at nothing. - [`s3-event-handler`](../../packages/s3-event-handler) indexes what Drive sync and other writers put in the bucket, and soft-deletes the rows of what they remove, through `BankDocumentService.reconcileObject`. Signed dashboard uploads are the exception, as above. - A report workflow whose index write throws fails its activity, so the miss is visible in Temporal rather than silent.