# A deposit or withdrawal is missing its ACH addenda **Last executed for real:** 2026-10-05, against prod, diagnosis only (ENG-3271). The deposits were `dp_CMBC2XD3fnrXAxzpW0HuZ` and `dp_gM3W4p97sg6wzNpWfjuZY`, Victor transactions `YEFA4P1SU5` and `U7LK59NLTJ`, posted 2026-10-05 16:21 UTC. The hold and the backfill script were written after that diagnosis: neither [Fix](#fix-backfill-the-entry-detail) nor [the hold branch](#an-inbound-ach-row-is-sitting-unreconciled) has been run from this runbook yet, and the step 1 query is adapted from the probe I ran and has not been run in this form. **Hold branch last executed for real:** 2026-10-28, against prod, diagnosis only (ENG-3568). `banktx_GPEmV8V7N0E0M43bL627l`, a Dash.fi Card ACH debit posted 15:31 UTC with no addenda or settlement date, was reported as not matching for recon. The cfx commands under [the hold branch](#an-inbound-ach-row-is-sitting-unreconciled) showed it held until 19:31 UTC, with its withdrawal `PENDING`. Victor completes an inbound ACH entry before it has finished describing it. In the first second after completion `GET /v2/transactions/{ref}` returns `addenda`, `company_id`, `company_entry_description`, `company_discretionary_data` and `settlement_date` as null. Hours later the same call returns them. Victor sends no webhook for the change, so the only way to see the enriched detail is to read it again. Observed on the two transactions above: null 0.6 to 0.8 seconds after completion, populated when read again about 7.5 hours later. A partner separately reported a lag of roughly 2 hours. How long Victor actually takes has not been measured; those readings only bound it. Two rails are affected, both on `dart` and both entries CFX does not originate: | Rail | `bank_transaction` row | Record that copies its entry fields | | ------------------ | ---------------------------------- | ----------------------------------- | | Inbound ACH credit | `type = CREDIT`, method `ACH` | The deposit's `deposit_quote.data` | | Inbound ACH debit | `type = DEBIT`, method `ACH_DEBIT` | `withdrawal.data` | ## Symptoms | What you see | Go to | | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | A partner reports a deposit or withdrawal with no addenda, for an entry that was sent with one | [Step 1](#step-1-what-victor-returned-at-ingest) | | A record from an inbound ACH entry has no company id, entry description or settlement date | [Step 1](#step-1-what-victor-returned-at-ingest) | | An inbound ACH credit or debit at Dart sits `POSTED` on `/bank-transactions`, under four hours old | [The hold](#an-inbound-ach-row-is-sitting-unreconciled) | | A reconciliation outcome's `reason` ends `reconciled without them.` or `reconciled without waiting.` | [Released without the fields](#the-hold-released-without-the-fields) | No alarm fires for any of these. ## Step 1: what Victor returned at ingest `bank_transaction.raw -> 'payload'` holds Victor's detail exactly as it was fetched when the row was inserted, and nothing rewrites it afterwards. A null there means Victor returned null at that moment. It rules out the mapping dropping the field. Fill in whichever id you have and leave the other placeholders as they are; they match nothing. ```bash cat > /tmp/ach_entry_detail.sql <<'SQL' \pset pager off \pset format unaligned \pset tuples_only on SELECT jsonb_pretty( jsonb_build_object( 'bank_transaction', bt.external_id, 'type', bt.type, 'writer', bt.raw ->> 'source', 'posted_at', bt.posted_at, 'created_at', bt.created_at, 'hold_until', bt.posted_at + interval '4 hours', 'reconciliation', bt.data -> 'reconciliation', 'victor_at_ingest', jsonb_build_object( 'addenda', p -> 'addenda', 'company_id', p -> 'company_id', 'company_entry_description', p -> 'company_entry_description', 'company_discretionary_data', p -> 'company_discretionary_data', 'settlement_date', p -> 'settlement_date' ), 'bank_transaction_keeps', jsonb_build_object( 'addenda', md -> 'addenda', 'companyId', md -> 'companyId', 'companyEntryDescription', md -> 'companyEntryDescription', 'companyDiscretionaryData', md -> 'companyDiscretionaryData', 'settlementDate', md -> 'settlementDate', 'secCode', md -> 'secCode' ), 'deposit', d.external_id, 'deposit_quote_keeps', jsonb_build_object( 'addenda', dq.data -> 'addenda', 'companyId', dq.data -> 'companyId', 'companyEntryDescription', dq.data -> 'companyEntryDescription', 'settlementDate', dq.data -> 'settlementDate' ), 'withdrawal', w.external_id, 'withdrawal_keeps', jsonb_build_object( 'addenda', w.data -> 'addenda', 'companyId', w.data -> 'companyId', 'companyEntryDescription', w.data -> 'companyEntryDescription', 'settlementDate', w.data -> 'settlementDate' ) ) ) FROM b2b.bank_transaction bt CROSS JOIN LATERAL ( SELECT bt.raw -> 'payload' AS p, bt.data -> 'methodData' AS md ) x LEFT JOIN b2b.deposit d ON d.external_id = bt.data -> 'reconciliation' ->> 'DEPOSIT' LEFT JOIN b2b.deposit_quote dq ON dq.id = d.deposit_quote_id LEFT JOIN b2b.withdrawal w ON w.external_id = bt.data -> 'reconciliation' ->> 'WITHDRAWAL' WHERE bt.external_id IN ('') OR bt.data -> 'reconciliation' ->> 'DEPOSIT' IN ('') OR bt.data -> 'reconciliation' ->> 'WITHDRAWAL' IN (''); SQL bun nx run db:psql:prod -- -q -f /tmp/ach_entry_detail.sql ``` `db:psql:prod` reads from the reader endpoint. The `\pset` lines keep the output one value per line; a plain aligned `SELECT` of these jsonb columns pads every row to the widest and is unreadable. | `victor_at_ingest.addenda` | `bank_transaction_keeps.addenda` | Linked record's `addenda` | Meaning | | -------------------------- | -------------------------------- | ------------------------- | ----------------------------------------------------------------------------------------------------------- | | null | null | null | Victor had not enriched the entry at ingest. Go to step 2 | | null | set | null | The row was filled later and the record was not. Go to [Fix](#fix-backfill-the-entry-detail) with `--ids` | | null | set | set | Already repaired. A stale read is the cache: [flush it](#flush-the-caches) | | set | null | any | Victor returned it and the row lost it. This runbook does not cover that; read the writer named by `writer` | No result means no bank transaction names that record under `DEPOSIT` or `WITHDRAWAL`. `writer` is `victor_webhook` or `victor_sync` for the rows this runbook covers. ## Step 2: what the webhook carried Optional; it confirms the webhook was as empty as the detail. Skip it for a `victor_sync` row, which no webhook wrote. The raw bodies are in Hookdeck, project `cfx-prod`, source `victor` (`src_gdak3kv2yiydwh` in [`imports.ts`](../../packages/backoffice-api/pulumi/hookdeck/imports.ts)). Hookdeck's body filter does not match Victor's payloads, so filter the source by a time window around `posted_at` and open the deliveries one by one until `id` is the Victor transaction id. An inbound ACH credit arrives as two `ach_inbound_credit` deliveries, `Pending` then `Success`. For the two transactions above both carried `addenda_details`, `company_id`, `entry_description`, `sec_code` and `settlement_date` as null, and Victor sent nothing further for either after `Success`. Where the webhook writer takes each entry field from when it inserts the row (`getAchEntryData` in [`VictorWebhookEventConsumer`](../../packages/backoffice-api/src/controllers/victor/VictorWebhookEventConsumer.ts)) decides what a null in the webhook costs: | Field on the row | Read from | | ----------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------- | | `addenda`, `settlementDate` | The GET detail only | | `effectiveEntryDate` | The GET detail, else the webhook body | | `senderName`, `achTraceId`, `victorTraceId`, `companyId`, `companyName`, `companyEntryDescription`, `companyDiscretionaryData`, `secCode` | The webhook body only | No code reads the webhook's `addenda_details`. ## Step 3: what Victor returns now The backfill script's dry run is the re-fetch. It reads `GET /v2/transactions/{ref}` for each row and prints what the row and its deposit or withdrawal would gain. Without `--apply` it writes nothing. Prerequisites: WARP connected ([WARP.md](../WARP.md)), `aws sso login --sso-session cfx`, and the RDS CA bundle, which bun does not trust by default: ```bash curl -sSo /tmp/rds-global-bundle.pem \ https://truststore.pki.rds.amazonaws.com/global/global-bundle.pem ``` ```bash AWS_PROFILE=workload-prod \ bun tools/secrets/run.ts --app temporal-worker --config prod -- env \ DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ NODE_EXTRA_CA_CERTS=/tmp/rds-global-bundle.pem \ bun packages/bank-transaction-reconciliation-services/scripts/backfillAchEntryDetail.ts \ --ids , ``` For dev, swap `workload-dev`, `-c dev` and `cfx-db-dev.cluster-cm87kdk8opyp.us-west-2.rds.amazonaws.com`. | `OUTCOME` | Meaning | | --------------- | ----------------------------------------------------------------------------------------------------- | | `WOULD FILL` | Victor now has fields the row or its record lacks. `FIELDS` names them per record. Go to the fix | | `UNCHANGED` | Victor has nothing the row and its record lack. Either still not enriched, or sent without an addenda | | `NOT ON VICTOR` | The detail GET returned 404 | | `SKIPPED` | The id names no row, or a row that is not an inbound Victor ACH entry on `dart` | | `FAILED` | The row threw. The message is in `FIELDS`, the run exits 1, and the other rows still ran | Five things about the command: - **SSM supplies the Victor credentials, not the database.** `cfx-temporal-worker` carries `VICTOR_BASE_URL`, `VICTOR_CLIENT_ID` and `VICTOR_CLIENT_SECRET` and no `DB_*`, which is why the command passes those by hand. - **Every run mints its own Victor token**, dry or not, with the environment's credentials. That can invalidate the token the running worker holds, so the worker's next Victor call can 401 once and re-mint. - **It dials the cluster endpoint, not the RDS Proxy.** `cfx_admin` gets `PAM authentication failed` at the proxy, which reads as a credential problem and is not one. - **Without `NODE_EXTRA_CA_CERTS` the dial fails `UNABLE_TO_GET_ISSUER_CERT_LOCALLY`**, after authenticating. - **Run it with plain `bun`**, never `tsx`. ## Fix: backfill the entry detail The same command with `--apply`: ```bash AWS_PROFILE=workload-prod \ bun tools/secrets/run.ts --app temporal-worker --config prod -- env \ DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ NODE_EXTRA_CA_CERTS=/tmp/rds-global-bundle.pem \ bun packages/bank-transaction-reconciliation-services/scripts/backfillAchEntryDetail.ts \ --ids , \ --apply ``` Per row it writes the entry fields the deposit quote or the withdrawal lacks, then the ones the bank transaction lacks. An unreconciled row gets only its own fields; reconciliation copies them when it runs. It is fill-only: a field a record already holds is never rewritten. Expect `FILLED` where the dry run printed `WOULD FILL`, and the line `The shared cache was not busted: …`. Then run the dry-run command again: every row reads `UNCHANGED`. Re-run step 1 and expect `bank_transaction_keeps` and the linked record to carry the fields. What the fill can and cannot restore, from [`achEntryDetail.ts`](../../packages/bank-transaction-services/src/achEntryDetail.ts): | Filled from Victor's detail | Never filled | | ---------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | | `senderName`, `victorTraceId`, `companyId`, `companyEntryDescription`, `companyDiscretionaryData`, `addenda`, `effectiveEntryDate`, `settlementDate` | `secCode`, `companyName`, `achTraceId` | The three in the right-hand column are set only when the row is inserted: all three from the webhook body on a `victor_webhook` row, and `achTraceId` alone from Victor's list item on a `victor_sync` row. A row inserted without them keeps them empty. ### Without `--ids` With no `--ids` the script selects `dart` rows that are an inbound ACH credit or debit, carry a Victor transaction id and have no addenda, newest posting first. `--limit` defaults to 25 and caps at 100, as does `--ids`. An entry that was sent without an addenda never leaves that selection and reads `UNCHANGED` on every run. An unbounded run therefore reaches the most recent rows and, once those fill the limit, never gets to older ones. Bound the run by posting time, dry first: ```bash AWS_PROFILE=workload-prod \ bun tools/secrets/run.ts --app temporal-worker --config prod -- env \ DB_HOST=cfx-db-prod.cluster-c1i9dhmqrdiz.us-west-2.rds.amazonaws.com \ DB_PORT=5432 DB_NAME=cfx DB_USER=cfx_admin AWS_REGION=us-west-2 \ NODE_EXTRA_CA_CERTS=/tmp/rds-global-bundle.pem \ bun packages/bank-transaction-reconciliation-services/scripts/backfillAchEntryDetail.ts \ --posted-after 2026-10-05T00:00:00Z --posted-before 2026-10-06T00:00:00Z --limit 100 ``` The first output line after the selection reads `Matched N bank transaction(s); considered M.` `N` above `M` means the window holds more rows than `--limit`; narrow it. Add `--apply` to the same command to write. Each row is one sequential Victor read. A row that fails is named at the end; re-run it with `--ids`. ### Flush the caches The script holds no Valkey client, so its writes clear only its own process. After an `--apply` that filled anything, cached reads of the records can serve the old value for up to 24 hours. Admin dashboard → System → Cache (`/cache`, needs `cache:w`). In the registry table press **Clear** on each of: - `bank-transaction-search` - `deposit-quote-search` - `deposit-search` - `withdrawal-search` The next deploy's `db:flush-caches` job clears them too. ### What the backfill does not do **It publishes no customer event.** Nothing changes status, so no deposit or withdrawal event goes out. A partner sees the fields only by reading the record again; tell them it is there. ## An inbound ACH row is sitting unreconciled Since the hold, an inbound ACH credit or debit at Dart whose detail has neither an addenda nor a settlement date is not reconciled straight away. Its workflow re-reads Victor's detail and waits, for at most four hours counted from the row's `posted_at`. While it waits the row reads `POSTED` and nothing downstream has happened: **for a credit the deposit does not exist yet**, so the customer's balance is not credited until the hold ends, and for a debit the withdrawal has not been finalized. The hold ends on whichever comes first: an addenda, a settlement date, or the deadline. An entry sent without an addenda is released by its settlement date. The admin bank-transaction sheet says so in its Reconciliation section: "Waiting on Victor's ACH entry detail" with the deadline, or "Released without Victor's ACH entry detail" once it passes and the row is still unreconciled. The quickest check from a laptop is the cfx CLI, with no psql and no Temporal API key: ```bash cfx prod get cfx prod wf bank-transaction-reconciliation- cfx prod sql "select k.key from b2b.bank_transaction b, jsonb_each(b.data -> 'methodData') k where b.external_id = '' order by 1" ``` `POSTED`, a `RUNNING` workflow and no `addenda` or `settlementDate` key is the hold. A `RUNNING` workflow past `posted_at` plus four hours, or a failed one, is not. `hold_until` in the [step 1](#step-1-what-victor-returned-at-ingest) query is the deadline. The workflow is the authority on whether the row is actually waiting: ```bash temporal workflow show \ --address platform-prod.qebt8.tmprl.cloud:7233 \ --namespace platform-prod.qebt8 \ --api-key "$(aws ssm get-parameter --name /cfx/secrets/temporal-worker/TEMPORAL_API_KEY --with-decryption --profile workload-prod --region us-west-2 --query Parameter.Value --output text)" \ --tls \ --workflow-id bank-transaction-reconciliation- \ --detailed ``` | In the history | Meaning | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `planBankTransactionReconciliation` returned `awaitAchEntryDetailUntilEpochMs` | The row is marked for the hold. The value is the deadline in epoch milliseconds | | The plan has no such key | Not held. The row was enriched at insert, or is not an inbound Victor ACH entry | | `refreshAchEntryDetail` activities separated by timers, status `Running` | It is waiting. Reads run at once, then after 30 seconds, doubling to a five-minute ceiling | | The plan is marked and no `refreshAchEntryDetail` ran | A worker still on the workflow code from before the hold ran it during a deploy and reconciled at once. `patched("ach-entry-detail-hold")` keeps that history replayable | | Status `Failed` | Not the hold. [ach-credit-not-reconciled.md](ach-credit-not-reconciled.md) step 2 reads the failure | | No execution exists | Reconciliation was never started for this row | Each `refreshAchEntryDetail` result is `{ enriched, outcome, fields }`: | `outcome` | Meaning | | --------------------- | ------------------------------------------------------------------------------------ | | `FILLED` | Victor's detail had fields the row lacked and they were written; `fields` names them | | `UNCHANGED` | Victor returned the detail with nothing new | | `NOT_FOUND_ON_VICTOR` | The detail GET returned 404. Counts as not enriched | | `ENRICHED` | The row already held an addenda or a settlement date. No Victor call was made | | `RECONCILED` | Something else reconciled the row in the meantime. The hold releases | `enriched: true` releases the hold. A read that fails its three attempts logs `ACH entry detail refresh failed; holding and re-reading.` and counts as not yet, so a Victor outage delays the row to its deadline and no further. A row in the hold needs no action. Nothing releases it early: **Run reconciliation** on `/bank-transactions` reports the row as started but attaches to the open run, and a fresh run would compute the same deadline from `posted_at`. ### The hold released without the fields The row reconciles anyway, and the outcome says so. The usual reason is followed by one of two sentences: | Ending of `reason` | Meaning | | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | | `Victor's ACH entry detail was still missing its addenda and settlement date when the hold window closed, so the row reconciled without them.` | It waited the full four hours | | `Victor's ACH entry detail had no addenda or settlement date, and the row posted longer ago than the hold window, so it was read once and reconciled without waiting.` | The run started more than four hours after `posted_at`, as on a re-drive of an old row | The worker logs the same sentence as a warning with the `bankTransactionId`. The deposit or withdrawal now exists without those fields and its event has gone out. Once Victor has the detail, run [the fix](#fix-backfill-the-entry-detail) with `--ids `. It fills the bank transaction and the record that copied it. ## Rollback The backfill has no undo. It overwrites nothing, changes no status and publishes nothing, so the only effect is fields appearing where there were none. The hold has nothing to roll back per row. Do not terminate a waiting run to hurry it: a terminated run is not retried, the row stays `POSTED`, and any run started for it afterwards computes the same deadline from `posted_at`. ## Prevention - The hold itself: [`bankTransactionReconciliationWorkflow`](../../packages/bank-transaction-reconciliation-services/src/bankTransactionReconciliationWorkflow.ts) and [`AchEntryDetailAdminService`](../../packages/bank-transaction-reconciliation-services/src/AchEntryDetailAdminService.ts). A partner reads the record when its event arrives, so the event is deferred until the record is complete. No event schema changed. [BANK_TRANSACTIONS.md § 3](../BANK_TRANSACTIONS.md#3-the-two-writer-pattern-which-path-reconciles-what) has the design. - [`AchDebitWithdrawalReconciliationService`](../../packages/withdrawal-services/src/ach-debit/AchDebitWithdrawalReconciliationService.ts) writes the entry fields to the withdrawal before it starts the finalize workflow, which is the run that publishes `COMPLETED`. - Tests: `bankTransactionReconciliationWorkflow.test.ts` and `AchEntryDetailAdminService.test.ts` in `bank-transaction-reconciliation-services/test/`, `achWithdrawalRailDetail.test.ts` in `withdrawal-services/test/`, and `depositQuoteAchEntryUpdate.test.ts` in `deposit-services/test/`. Known limits, none fixed: - The `PENDING` withdrawal event for an inbound ACH debit fires on Victor's `Pending` webhook, before settlement. The withdrawal has no addenda at that point. - The originated ACH credit withdrawal path ([`AchCreditWithdrawalReconciliationService`](../../packages/withdrawal-services/src/ach-credit/AchCreditWithdrawalReconciliationService.ts)) still writes its entry fields after starting the finalize workflow. - The four-hour cap is a choice, not a measurement. One observation was populated by 7.5 hours and one report says about 2; an entry Victor enriches later than four hours reconciles without its fields and needs the backfill. - `secCode`, `companyName` and `achTraceId` cannot be recovered from Victor's detail. - The backfill script cannot bust the shared caches and publishes no event.