Skip to content

feat(scripts): backfill security businesses from ingested executions - #4256

Draft
gilgardosh wants to merge 2 commits into
mainfrom
feat/security-businesses-backfill
Draft

feat(scripts): backfill security businesses from ingested executions#4256
gilgardosh wants to merge 2 commits into
mainfrom
feat/security-businesses-backfill

Conversation

@gilgardosh

Copy link
Copy Markdown
Collaborator

Step 5 of the per-security businesses plan. Stacked on #4255.

Ingestion (#4255) creates a business per security as executions arrive; the history that predates it
has none, and its trades still point at the general foreign-securities business.
yarn backfill:security-businesses closes that gap in two idempotent steps:

  1. A business per ISIN from poalim_securities_transactions, with a POALIM_SECURITY_KEY
    identifier per key seen reporting it, inheriting sort code / IRS code / country / tax category
    from the general business — the same thing ensureSecurityBusiness does.
  2. Re-point the trades whose description names exactly one resolvable key. Only transactions
    currently on the general foreign-securities business, and only non-fee rows.

Dry-run by default; --apply writes and --owner=<uuid> limits the run. It sets
app.current_business_id per tenant, as the server does per request, so it behaves the same whether
or not the connecting role bypasses RLS.

Verified end-to-end against a local scenario

Two Poalim keys reporting one ISIN, one execution with no ISIN, one plain trade, one fee row, one
description naming two securities:

Owner 00000000-…-0009a0
  securities created:      1
    · APPLE INC (AAPL)
  Poalim keys linked:      2
  transactions re-pointed: 1
  ⚠ 1 security key(s) report no ISIN — assign by hand: 1111111
  ⚠ 1 transaction(s) name more than one security and were left alone

Resulting counterparties — the trade moved, the fee row and the ambiguous one did not:

transaction is_fee counterparty
ניע"ז קניה 0005129523 f APPLE INC (AAPL)
ניעז עמ׳ תשלום FSEC PYMNT FEE 0005129523 t Foreign Securities
ניע"ז 0005129523 0007654321 f Foreign Securities

A second --apply created and re-pointed nothing (still one security business). The dry run reported
exactly what the apply then did. Scenario data removed afterwards.

yarn lint and yarn prettier clean.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a one-time backfill utility to align historical Poalim securities data with the new “business per security (ISIN)” model introduced during ingestion, and to re-point eligible legacy trades to the per-security business.

Changes:

  • Add scripts/backfill-security-businesses.ts to (1) create per-ISIN security businesses and link POALIM_SECURITY_KEY identifiers, then (2) re-point non-fee transactions currently on the general foreign-securities business when exactly one resolvable key is present in the description.
  • Wire the script as yarn backfill:security-businesses.
  • Add a changeset documenting the new backfill command and behavior.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
scripts/backfill-security-businesses.ts New backfill script implementing the two-step historical migration (create security businesses + repoint trades), with dry-run/apply modes.
package.json Adds backfill:security-businesses script entry.
.changeset/security-businesses-backfill.md Documents the new backfill command and its intended behavior/idempotency guarantees.
Suppressed comments (2)

scripts/backfill-security-businesses.ts:343

  • The UPDATE only filters by transaction id. Adding owner_id and the expected current business_id makes the repointing safer (avoids updating a transaction that was reassigned between the SELECT and UPDATE, and makes the intent explicit).
    if (apply) {
      await client.query(`UPDATE ${SCHEMA}.transactions SET business_id = $1 WHERE id = $2`, [
        businessId,
        transaction.id,
      ]);

scripts/backfill-security-businesses.ts:262

  • In dry-run mode, identifiersLinked is incremented for every Poalim key in the feed, even if that key is already linked from an earlier run or from ingestion. This can significantly over-report what “--apply” would actually create.
  const businessIdByPoalimKey = new Map<string, string>();

  for (const group of groups.values()) {
    let businessId = businessByIsin.get(group.isin);


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/backfill-security-businesses.ts Outdated
@gilgardosh
gilgardosh force-pushed the feat/security-businesses-automation branch from 2216167 to 372ead7 Compare August 20, 2026 09:45
@gilgardosh
gilgardosh force-pushed the feat/security-businesses-backfill branch from 9742d03 to 4442f0b Compare August 20, 2026 09:47
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 09:48 — with GitHub Actions Inactive
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 09:48 — with GitHub Actions Inactive
@gilgardosh
gilgardosh force-pushed the feat/security-businesses-automation branch 2 times, most recently from f55d378 to 4b5729a Compare August 20, 2026 13:16
Base automatically changed from feat/security-businesses-automation to main August 20, 2026 13:34
@gilgardosh
gilgardosh force-pushed the feat/security-businesses-backfill branch from 4442f0b to fed3740 Compare August 20, 2026 13:38
@gilgardosh
gilgardosh changed the base branch from main to feat/security-businesses-plan August 20, 2026 13:38
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:38 — with GitHub Actions Inactive
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:38 — with GitHub Actions Inactive
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:38 — with GitHub Actions Inactive
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:38 — with GitHub Actions Inactive
gilgardosh and others added 2 commits August 20, 2026 16:39
Creates a business per ISIN out of poalim_securities_transactions with a
POALIM_SECURITY_KEY identifier per key, then re-points non-fee transactions
that currently point at the general foreign-securities business and whose
description names exactly one resolvable key.

Dry-run by default; --apply writes, --owner limits to one tenant. Idempotent,
and reports what it deliberately left alone.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Addresses review feedback: the three inserts a security takes ran without a
transaction, so a failure part-way through left a business that is not a
security - invisible to the ISIN lookup and duplicated by the next run, which
is exactly the guarantee the script makes.

Also normalizes the currency through formatCurrency. businesses_securities
.currency_code is the accounter_schema.currency enum since #4252, and the
executions report the feed's own Hebrew labels, so the insert would have
failed outright on the first real security.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@gilgardosh
gilgardosh force-pushed the feat/security-businesses-backfill branch from fed3740 to cb23ed0 Compare August 20, 2026 13:40
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:40 — with GitHub Actions Inactive
@gilgardosh
gilgardosh temporarily deployed to accounter-fullstack August 20, 2026 13:40 — with GitHub Actions Inactive
@github-actions

Copy link
Copy Markdown
Contributor

🚀 Snapshot Release (alpha)

The latest changes of this PR are available as alpha on npm (based on the declared changesets):

Package Version Info
@accounter/client 0.1.0-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/green-invoice-graphql 0.8.7-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/hashavshevet-mesh 0.2.13-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/israeli-vat-scraper 0.1.13-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/modern-poalim-scraper 0.11.0-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/payper-mesh 0.2.13-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/scraper-app 0.0.3-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/server 0.2.0-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/shaam-uniform-format-generator 0.2.7-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎
@accounter/shaam6111-generator 0.1.9-alpha-20260820133947-eadbfa3872eec9cb96ce0dc92561f34d6eaf0c2b npm ↗︎ unpkg ↗︎

@gilgardosh
gilgardosh changed the base branch from feat/security-businesses-plan to main August 20, 2026 13:49
@gilgardosh
gilgardosh marked this pull request as draft August 20, 2026 13:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants