add analysis and summary of trade data approach for #527

[bl-young]

cc:
Closes: #527

Adds analysis and write-up associated with exploration of trade data.

Summary

Trade data for F04000 / F05000 — intended plan

Decision: Use Option 1 — Census trade in goods + BEA trade in services as the primary annual extract. Scale / constrain national totals to BEA ITA goods+services controls. BACI is out of scope for this F040/F050 work.

Background and 2017 test results: trade_data_source_options_527.md. Longer plan notes (annual summary IO check, compare() results): plan_527_long.md.

---

What we are building

Annual initial estimates for Use-table F04000 (exports) and F05000 (imports) for nowcast years, aligned as closely as practical with the 2017 detailed SUT/MUT (exports PUR; import valuation consistent with Supply CIF / MCIF), with national goods+services totals controlled to ITA.

Source roles

Role	Source	Use
Primary extract (goods)	Census International Trade (NAICS-6)	Imports CIF (GEN_CIF_YR) to match Supply CIF; exports FAS-family (ALL_VAL_YR). Customs (GEN_VAL_YR) available if combining with services needs a no-double-count diagnostic.
Primary extract (services)	BEA IntlServTrade	Imports and exports by type of service
National totals control	BEA ITA Tables 2.1 / 3.1 (+ NIPA foreign linkage as needed)	Scale or residual-constrain combined Census+BEA to ITA goods+services export/import totals
Sector bridge	USEEIO import-EF concordances (+ bedrock NAICS↔BEA Detail)	NAICS → BEA Detail (strong); service type → BEA Detail (thin today — extend)
2017 truth / residuals	Bedrock Use F040/F050, Supply MCIF	Benchmark shape and specials (S00300, S00900, wholesale/margins)

Not used: CEPII BACI / ceda BACI — FOB, goods-only, share-oriented; not needed for US F040/F050 levels or ITA-controlled totals.

Where to pull implementation from

Piece	Pull from	Notes
Goods + services download (imports and exports)	flowsa imports — Census_USATrade, BEA_IEA	Preferred. Not in bedrock or flowsa master yet.
ITA control totals	BEA ITA 2.1 (goods) + 3.1 (services); optional NIPA foreign-transactions linkage	No bedrock ITA loader yet — add or call BEA API for G+S totals
Legacy imports-only reference + concordances	USEEIO import_emission_factors/ — especially concordances/Census_to_useeio2_sector_concordance.csv, BEA_service_to_useeio2_sector_concordance.csv	Goods map ~97% of Census import value; service map only ~10 API types
Broader NAICS↔BEA Detail	bedrock/utils/mapping/naics/NAICS_to_BEA_Crosswalk_2017.csv	Already in bedrock
2017 F040/F050 / MCIF	Bedrock IO extract / Use and Supply tables	Benchmark for structure and specials

Do not treat USEEIO’s download_imports_data.py as the production extract — it is imports-only. Prefer flowsa imports FBAs, then map with USEEIO/bedrock concordances.

Pipeline (conceptual)

Census goods (CIF imports, FAS exports) ─┐
                                         ├─→ map to BEA Detail ─→ commodity vectors
BEA IntlServTrade (imp + exp) ───────────┘         │
                                                   ├─ specials / margins from Use or residual policy
                                                   ├─ scale / constrain national totals to ITA G+S
                                                   └─ nowcast years via same recipe (#528)

Known gaps to close before “recreates 2017”

1. CIF goods + full services can overshoot Use/MCIF (~12% in 2017 tests) — ITA totals control (and/or customs vs CIF) addresses the national overshoot; keep CIF when targeting Supply MCIF unless the control step replaces levels.
2. Service→Detail map is thin — export Use has large holes (533000 IP-like, travel, etc.); extend concordance.
3. Special IO / margin cells (S00300, S00900, wholesale) — explicit hold-from-Use or residual policy.
4. Exports PUR — FAS ≠ PUR; margins/BOP bridge still open.
5. Wire flowsa imports + ITA totals into bedrock’s extract/FBA path, then FBA/FBS → nowcast.

Implementation order (#528)

1. Vendor/merge flowsa Census_USATrade + BEA_IEA into bedrock’s extract path; add ITA 2.1/3.1 totals for the control step.
2. Apply goods + services → BEA Detail maps; document valuation, specials, and ITA scale/residual rules.
3. Prove 2017 reconstruction: commodity shape vs Use F040/F050 (and MCIF as needed); national totals vs ITA.
4. Hook into nowcast initial Y for non-benchmark years.

[WesIngwersen]

@bl-young I've decided it would be more prudent to have a base branch for this nowcast work. Pull all intermediate steps in to this branch rather than main. The pull into main will be the very end. We will keep nowcast fresh with main updates by occassionally pulling in main.

[WesIngwersen]

Here is something I wasn't clear enough about. I think we should just gear this toward SUT Supply Use 2017 as the target diagnostic. For imports this means we need to target not F0500 but the following cols from the Use table: MCIF (imports in CIF), MADJ (CIF/FOB Adjustments on Imports) as well as MDTY (Import duties). A following step beyond the scope of this work will be to convert this into MUT but that will be done with the full SUT -> MUT conversion.