# A customer document is in S3 but not in `b2b.customer_document` **Initial backfill executed 2026-10-08:** dev 7 rows across 5 customers, prod 74 rows (73 files, 1 folder) across 34 customers, none skipped; both re-checks clean. `b2b.customer_document` indexes the files under `s3://cfx-customer-documents-/customer-documents//`: one row per file or folder. S3 holds the bytes, the table holds the listing. A customer's documents are listed from the table, so a file with no row is in S3 and on no page. `s3-event-handler` also writes rows: S3 tells it about every object created or deleted in the bucket, and within seconds it inserts the row for a new file (and the folder rows above it) or soft-deletes the row of a file that is gone. Files `file-sync` copies in from Google Drive are indexed that way. This runbook is for the cases the handler leaves, or fails on. Each type has its own prefix under the customer's root: `CUSTOM` files sit at the root itself, and `pricing-schedules/`, `logos/` and `terms/` hold the other three types. The run indexes each type from its own prefix and leaves those three subtrees out of `CUSTOM`. The terms and pricing schedule PDFs are read from S3 by their fixed keys, not from the table, so a missing row never hides one from the terms flow or the pricing schedule card. It only keeps the file off the document listings. ## Symptoms → cause | What you see | Cause | | ---------------------------------------------------------- | ----------------------------------------------------------------- | | The table is empty after its migration deployed | The initial backfill has not run | | A file written before the index deployed has no row | Same: the handler only sees objects created after its rule exists | | A Drive file has no row minutes after it reached S3 | The handler failed on it; look in the DLQ | | `cfx-s3-event-handler-dlq-age-` fired | An event failed three times, or EventBridge could not deliver it | | A dashboard upload the browser never completed has no row | The handler leaves signed uploads to `completeUpload`; run Step 2 | | A file copied within S3 has no row | The copy keeps its source's `uploaded-by` mark; run Step 2 | | A file written by an app still on the old task or Lambda | That app had not picked up the deploy; see below | | A pricing schedule upload failed after its file reached S3 | The S3 put succeeded and the index write threw | | A document lists, but opening it says it is not found | Its object is gone and the row was not soft-deleted; see below | All but the last are repaired the same way. The run reads every object version under each customer's prefix and inserts the rows that are missing. It never updates or deletes a row and never writes to S3, so it is safe to run again at any time. `file-sync` (the `cfx-self-hosted-services-` EC2 host, [packages/file-sync](../../packages/file-sync/README.md)) bisyncs the bucket with Google Drive every two minutes, both ways. A file someone adds in Drive reaches S3 and the handler indexes it; a file deleted in Drive is deleted from S3 and the handler soft-deletes its row. The bisync runs without `--track-renames`, so a move or rename in Drive arrives as a delete plus a fresh upload: the old row is soft-deleted and a new one inserted. The handler never indexes an object that carries `uploaded-by` metadata, because that is a dashboard user's signed upload and its row belongs to `completeUpload`; a row written first would make `completeUpload` refuse the user's upload. Two consequences. An upload the browser never completes has no row until the backfill runs, which does index marked objects (the next paragraph says how it attributes them). And an object copied within S3 keeps its source's `uploaded-by` mark, for example the copy a bisync conflict rename makes or a restored version, so it also waits for the backfill. `created_at` comes from the object's first S3 version and `updated_at` from its latest, so a backfilled row carries the file's real dates, not the time of the run. Every object the run would insert is HEADed for its `x-amz-meta-uploaded-by` metadata, which a signed upload carries. A file a dashboard user uploaded but never completed gets `updated_by_user_id` set to that user, a member of the customer's workspace. One marked with someone who is not a member of the customer is not indexed and counts in `skipped`; it stays in S3 for you to inspect. An unmarked object, written by CFX staff, Drive sync, a workflow or by hand, gets null, which reads as "CFX". ## Before you start - Cloudflare WARP on ([docs/WARP.md](../WARP.md)); the database is not reachable without it. - An admin SSO profile. The run lists object versions, which needs `s3:ListBucketVersions`, and no app's role is granted that on this bucket. - `rds-global-bundle.pem` from AWS for `NODE_EXTRA_CA_CERTS`. - Valkey, over the same WARP tunnel. The listings are cached for 24 hours in the `customer-document-search` namespace ([CACHING.md](../CACHING.md) §9), and a run that inserts clears it. The `VALKEY_*` values below are the ones the post-deploy flush uses (the `Flush non-auth Valkey caches` step in [deploy.yml](../../.github/workflows/deploy.yml)), with the admin tier's user `cfx-admin-` in place of the CD one. The script refuses to start without them, `--check` included. - A tree at the commit the stage is running (`git archive | tar -x -C `, then `bun install --frozen-lockfile --ignore-scripts`). Run from a `main` that carries an undeployed migration and the first read fails with `column "…" does not exist` (`42703`). ## Step 1: see what is missing `--check` reads S3 and the table and writes nothing. Dev: ```bash aws sso login --sso-session cfx NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-dev \ env DB_HOST=cfx-db-dev.cluster-cm87kdk8opyp.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ CUSTOMER_DOCUMENTS_S3_BUCKET=cfx-customer-documents-dev \ VALKEY_HOST=master.cfx-cache-dev.mueox9.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-dev VALKEY_USERNAME=cfx-admin-dev \ bun packages/customer-services/scripts/backfillCustomerDocuments.ts --check ``` Prod: ```bash aws sso login --sso-session cfx NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-prod \ env DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ CUSTOMER_DOCUMENTS_S3_BUCKET=cfx-customer-documents-prod \ VALKEY_HOST=master.cfx-cache-prod.h4koef.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-prod VALKEY_USERNAME=cfx-admin-prod \ bun packages/customer-services/scripts/backfillCustomerDocuments.ts --check ``` Expected shape, one line per customer (the ids and counts here are made up), and exit code 1 while anything is pending: ``` cust_ZQuernKL7nUH1c6cqNiVm: 14 file(s) in S3, would insert 14, skipped 0; 2 folder(s), would insert 2. cust_4kTnS0d8Wd3Jx1bq2PzLm: 0 file(s) in S3, would insert 0, skipped 0; 0 folder(s), would insert 0. ``` `skipped` counts keys whose path the index cannot hold, such as an empty segment (`a//b.pdf`), and objects whose uploader is not a member of the customer. They stay in S3 and out of the listing. Add `--customer ` to run one customer. ## Step 2: insert the missing rows The same command without `--check`. Prod: ```bash NODE_EXTRA_CA_CERTS=/path/to/rds-global-bundle.pem AWS_PROFILE=workload-prod \ env DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ CUSTOMER_DOCUMENTS_S3_BUCKET=cfx-customer-documents-prod \ VALKEY_HOST=master.cfx-cache-prod.h4koef.usw2.cache.amazonaws.com \ VALKEY_REPLICATION_GROUP_ID=cfx-cache-prod VALKEY_USERNAME=cfx-admin-prod \ bun packages/customer-services/scripts/backfillCustomerDocuments.ts ``` The lines now read `inserted` with the same counts Step 1 reported, followed by one more (the count is made up): ``` Cleared 9 cached customer document key(s). ``` That line is the cache bust. For a customer whose listings were cached and which gained rows, the run also prints `[valkey-cache] invalidateTag failed` with `no permissions to run the 'hdel' command`. That is expected and harmless here: the service busts by tag, and the admin tier's Valkey user may `DEL` but not `HDEL` (`cacheBustAccessString` in [packages/aurora/pulumi/index.ts](../../packages/aurora/pulumi/index.ts)). The script therefore clears the whole namespace itself, which that user is allowed to do. If the `Cleared` line is missing, the run failed before it: fix what it reports and run Step 2 again. ## Step 3: verify Run Step 1 again. Every line reads `would insert 0` and the exit code is 0. A Drive sync that lands between the two runs can leave a line above 0; run Step 2 again for that customer. ## A deploy that has not reached every writer `admin-dashboard` writes customer documents (uploads and pricing schedules) and `customer-dashboard` reads them. A dashboard Lambda that started before the deploy keeps writing to S3 without a row until it is replaced. Confirm the artifact, not the merge: the dashboard Lambda's `LastModified`. Then run Steps 1 to 3 once more to pick up whatever the old code wrote in between. ## The handler's dead-letter queue A failed invocation is retried twice by Lambda, then parked in `cfx-s3-event-handler-dlq-`; `cfx-s3-event-handler-dlq-age-` fires while it holds a message. The shared procedure for reading and replaying a parked message is [sns-consumer-dlq-parked.md](sns-consumer-dlq-parked.md) § EventBridge consumers. For this handler, resolve the queue URL once. Prod: ```bash Q=$(AWS_PROFILE=workload-prod AWS_REGION=us-west-2 aws sqs get-queue-url \ --queue-name cfx-s3-event-handler-dlq-prod --query QueueUrl --output text) ``` Dev: ```bash Q=$(AWS_PROFILE=workload-dev AWS_REGION=us-west-2 aws sqs get-queue-url \ --queue-name cfx-s3-event-handler-dlq-dev --query QueueUrl --output text) ``` 1. Read the parked messages without consuming them (use the profile that matches `$Q`): ```bash AWS_PROFILE=workload-prod AWS_REGION=us-west-2 aws sqs receive-message \ --queue-url "$Q" --max-number-of-messages 10 --visibility-timeout 0 ``` 2. Find the error in `responsePayload.errorMessage`, and the bucket and key in `requestPayload.detail`. Fix the cause (a deploy, a grant) before replaying. 3. Replay `requestPayload` with `aws lambda invoke --invocation-type Event`, following [sns-consumer-dlq-parked.md](sns-consumer-dlq-parked.md) § EventBridge consumers. Replaying is the repair and is always safe: a key already in step is `UNCHANGED`. Steps 1 to 3 above stand in only for a parked `Object Created` (`detail-type`, with the bucket in `detail.bucket.name`), and only for the row: they insert missing rows, publish no `customer.document.created` event, and never update or delete a row. A parked `Object Deleted` is not repaired by them (the live row stays and the file opens as not found): replay it, or delete the document from the admin dashboard. The handler logs one JSON line per event to `/aws/lambda/cfx-s3-event-handler-`: `{"message":"s3 object event","id":…,"bucket":…,"keySha256":…,"outcome":…,"reason":…}`. The key is never logged, because keys hold customer short names and file names; `keySha256` is the lowercase hex SHA-256 of the key. To find a file's line: ```bash printf %s '' | shasum -a 256 ``` Then search the log group for that hash. `outcome` is `INSERTED`, `DELETED`, `UNCHANGED` or `SKIPPED`; a skip carries its `reason`: `NOT_A_DOCUMENT_KEY`, `OWNER_NOT_FOUND` or `SIGNED_UPLOAD`. If the Lambda's `Throttles` alarm fires, more than 20 events arrived at once, for example a large folder deleted in Drive. Lambda holds and retries throttled events, so nothing is lost; act only if the DLQ alarm follows. ## Limitation: a lost event The handler writes the row, then publishes `customer.document.created` or `customer.document.deleted` on the internal event bus. If publishing fails after the row is written, Lambda retries the invocation, finds the row already in step and does not publish again. That one event is lost; the row is correct. ## A live row whose object is gone A file deleted in Drive is soft-deleted by the handler. Rows orphaned before the handler deployed (an object already gone, its row live) are deleted from the admin dashboard's customer page, which soft-deletes the row and finds no object to remove. ## Rollback There is nothing to roll back in S3; the run does not write there. It only adds index rows that mirror objects which exist. If an object should not be there, delete the document from the admin dashboard, which removes the object and soft-deletes its row. ## Prevention - [`CustomerDocumentService.create`](../../packages/customer-services/src/CustomerDocumentService.ts) writes the row in the same call as the S3 put, S3 first, so a failure leaves an unindexed object this run repairs and never a row pointing at nothing. The shared engine is [`DocumentIndex`](../../packages/document-services/src/DocumentIndex.ts), the same one bank documents use ([bank-document-not-indexed.md](bank-document-not-indexed.md)). - [`s3-event-handler`](../../packages/s3-event-handler) indexes what Drive sync and other writers put in the bucket, and soft-deletes the rows of what they remove, through `CustomerDocumentService.reconcileObject`. Signed dashboard uploads are the exception, as above.