Repository navigation
Description
In order to catch mistakes in our federal math on returns we have not checked by hand, we can compare our engine against PolicyEngine-US in two ways: compare the numbers it produces, and compare the logic it uses for each federal line.
PolicyEngine-US is an open-source tax and benefits calculator written in Python. Policy researchers use it to model federal and state tax. It is independent of us and of the IRS forms, so when it agrees with our engine to the dollar that is strong evidence, and when it disagrees one of the two has a bug worth finding.
Part 1: compare the numbers
The idea comes from opentax-engine (Invaro). They generate 572 simple returns by combining wages, filing status, children, interest, capital gains and itemized deductions, run both engines, and sort the differences into buckets. Every difference has to be explained with an IRS citation or it fails the build.
Example outcome:
Scenarios run: 480
Agree to the dollar: 452 (94.2%)
Within $1 (rounding): 9
Explained, with citation: 19
Unexplained: 0 ← anything here fails the test
Explained differences
- Child tax credit threshold for a qualifying surviving spouse: we use $200,000,
PolicyEngine uses $400,000. IRS Schedule 8812 instructions say $200,000. (19 scenarios)
Part 2: compare the logic, line by line
PolicyEngine implements each federal line as a small, readable Python formula with its own source file, for example the formula for the taxable amount of social security benefits or for the child tax credit phase-out. We can put our binding for the same line next to their formula and read the two together. This finds bugs the number comparison cannot reach: cases neither of us generated, thresholds that only matter at high income, and places where we both happen to be wrong the same way.
Example outcome: a short review note per form we support, in the form's folder:
Schedule D, line 16 vs PolicyEngine `capital_gains` / `net_capital_gain`
- Same: short-term and long-term netting order, $3,000 loss cap ($1,500 MFS).
- Different: PolicyEngine carries the unused loss forward itself; we expect it as a fact. OK for us.
- Found: we did not treat collectibles gain at 28%. Fixed in #NN.
Two notes. Invaro found PolicyEngine wrong twice, so a disagreement is a lead, not a verdict. And this is federal only. PolicyEngine's California coverage is thinner than ours.
Refs
Activity
[-]Engine: differential test of federal lines against PolicyEngine-US[/-][+]Compare our federal results against PolicyEngine as a second opinion[/+][-]Compare our federal results against PolicyEngine as a second opinion[/-][+]Compare our federal results and line logic against PolicyEngine as a second opinion[/+]captaincole commentedon Nov 1, 2026
Part 2, first pass: line logic vs PolicyEngine, form by form
Setup: a sparse checkout of policyengine-us 2.88.9 (commit
10c7f5a), itsvariables/gov/irsformulas and theparameters/gov/irsvalues in force for tax year 2025, read next to each form'sbindings.ts. Federal only. Three forms have no PolicyEngine counterpart and were skipped: Form 8949 (per-lot listing), Schedule B (sums only) and Schedule H (household employment). Every lead was then given a second, skeptical pass: is it right against the 2025 IRS text in the corpus and our code, and can a wrong number actually print today? A disagreement with PolicyEngine is a lead, not a verdict; the second pass decides.What agrees: every 2025 constant both sides touch. Rate schedule brackets and the tax table, standard deduction and the 65+ amounts, the 0/15/20% capital gain ceilings and the QDCG worksheet stacking, CTC $2,200 / $1,700 / phase-out thresholds, HSA limits, IRA and student loan phase-outs, the SALT cap and phase-down (except MFS, below), Additional Medicare and NIIT thresholds, QBI thresholds and phase-in range, and the 2025 AMT exemption and phase-out numbers.
Leads on our side
must_file_schedule_1, so the fallback is the normal path. Correction to the first note: a missing line 13 makes AGI too high; a missing line 8f makes it too low.Items 3, 4, 5 and 6 share one shape: a blank or unsupported line is read as zero by the next line, so the error is silent. That is the pattern to fix once, not four times.
On PolicyEngine's side
Next
Notes by form follow, one comment each.
captaincole commentedon Nov 1, 2026
Form 1040 vs PolicyEngine
adjusted_gross_income,taxable_income,income_tax_main_rates,capital_gains_tax,income_tax_capped_non_refundable_credits,ctc,income_taxLines 1–16
Lines 17–38
income_taxleaves out self-employment tax, Additional Medicare Tax and household employment taxes (they sit in its payroll-tax total); our line 24 follows the form. Withholding (25a–c), estimated payments (26) and excess FICA (Schedule 3 line 11) are inputs on both sides. Our lines 25d and 33 print 0, not blank, when nothing was withheld (cosmetic). Line 25c takes only 8959 line 24 (input gap).ctc/phase_out/threshold.yaml, SURVIVING_SPOUSE). The 2025 Schedule 8812 instructions, "Limits on the CTC and ODC", give $400,000 to married filing jointly only; IRC §24(h)(3) says "in the case of a joint return". Already filed upstream as Federal CTC phase-out threshold for surviving spouses is $400,000 from 2018; 24(h)(3) gives $200,000 PolicyEngine/policyengine-us#9703 with a fix in Fix the 2021 CTC split and surviving-spouse phaseout threshold PolicyEngine/policyengine-us#10306. Nothing on our side; line 19 is still an input gap.captaincole commentedon Nov 1, 2026
Schedule 1 vs PolicyEngine
above_the_line_deductions,adjusted_gross_income,irs_gross_incomemiscellaneous_incomeinput. Prizes technically belong on line 8i. Our HSA limit prorates by coverage months; PolicyEngine takes the deduction as an input. Our totals (lines 9, 10, 25, 26) count an unsupported component line as 0; PolicyEngine sums every source. Worth a look.engine/src/filingInfoResolvers.ts). A filer with an HSA deduction (8889 line 13) or taxable HSA distributions (8889 line 16) and no 1099-MISC gets no Schedule 1, so 1040 lines 8 and 10 stay blank. Form 8889 instructions, line 13 ("enter on Schedule 1, Part II, line 13") decide it. Lead Split the engine and the forms out of the agent #5.must_file_schedule_1ormust_file_8889, so the fallback is the normal path, and Cade does record HSA contributions after a 1099-SA. A missing line 13 makes AGI too high; a missing line 8f makes it too low. Schedule 2's resolver already derives its must-file from 8959, 8960 and Schedule H, so the same shape fits here: file Schedule 1 when 8889 files or box 3 > 0. No golden changes (jordan already files it; marcus has a code W but no coverage facts).loss_ald, adoption and savings-bond exclusions.captaincole commentedon Nov 1, 2026
Schedule 2 vs PolicyEngine
alternative_minimum_tax,additional_medicare_tax,net_investment_income_taxincome_tax_before_credits, NIIT intoincome_tax_before_refundable_credits, and Additional Medicare intoemployee_payroll_tax.unreported_payroll_tax,qualified_retirement_penalty,recapture_of_investment_credit).captaincole commentedon Nov 1, 2026
Schedule 3 vs PolicyEngine
income_tax_non_refundable_credits,income_tax_capped_non_refundable_credits,energy_efficient_home_improvement_creditexcess_payroll_tax_withheldis an input with no formula, the premium tax credit sits outside its refundable list, and lines 10, 12 and 13a–13z have no counterpart. Lines 6a–6c, 6g–6l and 6z likewise.captaincole commentedon Nov 1, 2026
Schedule A vs PolicyEngine
salt_cap,salt_deduction,medical_expense_deduction,charitable_deduction,interest_deductioncaptaincole commentedon Nov 1, 2026
Schedule D vs PolicyEngine
net_capital_gain,capital_gains_tax,filer_loss_limited_net_capital_gainscaptaincole commentedon Nov 1, 2026
Form 8889 vs PolicyEngine
health_savings_account_aldhsa.coverage_monthsis a count. Proration is the right default, and the rule belongs behind a decision fact (same shape asdecisions.scope.must_file_8889). The catch-up gap is real: rule 6 puts it on line 3 for an unmarried 55+ filer, line 7 is only the married family-coverage case, and our line 3 never adds it. No scenario is 55+ with an HSA. Smaller: the 1099-SA sum does not check box 5 (HSA vs MSA); one-line filter, not reachable today.captaincole commentedon Nov 1, 2026
Form 8959 vs PolicyEngine
additional_medicare_tax,additional_medicare_tax_withheldmust_file_8959, so it is missed today. Out of scope for a single filer. Fix: use the status threshold when the status is known, else keep $200,000.captaincole commentedon Nov 1, 2026
Form 8960 vs PolicyEngine
net_investment_income_tax,net_investment_income,niit_magicaptaincole commentedon Nov 1, 2026
Form 8995-A vs PolicyEngine
qbid_amount,qualified_business_income_deduction,section_199a_net_capital_gaincaptaincole commentedon Nov 1, 2026
Finding 1: the capital loss limit is missing (1040 line 7a, Schedule D line 21)
Our position: We agree. This is a real bug and the most important one in the list. A filer with a $20,000 net capital loss gets the full $20,000 off AGI today. The limit is $3,000 ($1,500 if married filing separately). The rest carries to next year.
Why it happens: 1040 line 7a copies Schedule D line 16 (
engine/src/federal/1040/bindings.ts). Schedule D line 21 is still a todo, so it prints blank. The todo note says we had no "smaller of" rule. Bindings are plain functions now, so that reason no longer applies.Fix:
1.line.21_allowable_loss: when line 16 is below zero, the smaller of the loss or the cap for the filing status. Otherwise blank.line.7a: line 21 when line 16 is a loss, line 16 when it is a gain.Priority: next engine PR.
Test: a new scenario with a loss over $3,000. No locked scenario has a net loss, so no golden changes.
captaincole commentedon Nov 1, 2026
Finding 2: SALT cap phase-down for married filing separately (Schedule A line 5e)
Our position: We agree the math is wrong for MFS, but it can wait. Our code halves the $40,000 cap first and then takes the full 30% reduction. The worksheet in the Schedule A instructions reduces $40,000, floors the result at $10,000, and halves it only at the end. Example: MFS, MAGI $300,000, line 5d $25,000. The worksheet gives $12,500. We give $5,000.
A second, smaller problem affects every filer: worksheet line 7 (30% of the excess) is not rounded. MAGI $500,001 prints $39,999.70 where the form wants $40,000.
Fix: in
0.line.5e_salt_cap, cap = at least $10,000 of ($40,000 minus the rounded 30% reduction). For MFS, halve that cap and round. UseroundDollarson the 30% line.Priority: later. MFS is out of scope for now. Fix the rounding the next time this file changes.
Test: one unit test per case: MFS in the phase-down, and a single filer just over $500,000. Only marcus files Schedule A, and his numbers do not change.
captaincole commentedon Nov 1, 2026
Finding 3: Form 8959 line 6 reads a blank line 5 as zero
Our position: We agree it is a bug, but it cannot print a wrong number today. With no filing status, line 5 (the threshold) is blank. Line 6 treats it as $0, so line 7 taxes every Medicare wage dollar. We reproduced it: marcus without a status shows $2,192 instead of $392.
It stays hidden because a missing filing status puts the whole return in pending. The UI and the documents show nothing until the status is set. If that gate ever changes, the wrong number shows up.
Fix:
0.line.6_excess_medicare_wagesreturns blank when line 5 is blank. This is the same pattern 1040 line 15 uses. We could also list the filing status as a missing input inresolve8959Info. Line 22 does not have this bug.Priority: later, but cheap. Pair it with finding 4, because both are a blank line read as zero.
Test: a unit test with no filing status that expects line 6 and line 7 to stay blank.
captaincole commentedon Nov 1, 2026
Finding 4: no zero floor on Form 8960 line 12
Our position: We agree. Fix it now. The printed form says "If zero or less, enter -0-" on line 12. Our catalog label dropped that part, so the binding never floors. A net capital loss bigger than interest plus dividends gives a negative net investment income, and line 17 becomes a negative tax.
This does not depend on finding 1. Even with a correct $3,000 cap, $1,000 of interest and a capped loss give line 12 = −$2,000 and line 17 = −$76.
Fix: wrap
0.line.12inatLeast(0, …)inengine/src/federal/8960/bindings.ts. Also restore the full label in the catalog.Priority: next engine PR, with finding 1.
Test: the finding 1 scenario, if it has interest income and files Form 8960. Only jordan files 8960 today, and his line 12 is positive, so no golden changes.
captaincole commentedon Nov 1, 2026
Finding 5: Schedule 1 does not turn on for an HSA filer
Our position: We agree. Fix it now. Only 1099-MISC box 3 turns Schedule 1 on. An HSA filer with no 1099-MISC files Form 8889, but the deduction on 8889 line 13 never reaches Schedule 1 line 13. AGI comes out too high, and nothing tells the user. A taxable HSA distribution (line 8f) is lost the same way, and that makes AGI too low.
Cade's decision table has no
must_file_schedule_1key, so the fallback rule is the normal path, not an edge case.Fix: in
resolveSchedule1Info(engine/src/filingInfoResolvers.ts), file Schedule 1 when Form 8889 must be filed or 1099-MISC box 3 is above zero.resolveSchedule2Infoalready derives its must-file from other forms the same way.Priority: next engine PR.
Test: a scenario with an HSA and no 1099-MISC. Check that 1040 line 10 carries the deduction. Jordan already files Schedule 1, so his numbers do not change.
captaincole commentedon Nov 1, 2026
Finding 6: capital gain distributions drop out when Schedule D is filed (Schedule D line 13)
Our position: We agree. Fix it with finding 1. When a filer has 1099-DIV box 2a and also files Schedule D, the box 2a amount goes nowhere. Line 13 is a todo, so it prints blank. The 1040 line 7a exception (distributions only, no Schedule D) turns off. The income disappears, and no pending input warns about it.
This pairing is common for our target user: a brokerage account with fund distributions and some trades.
Fix: add capital gain distributions to the Schedule D filing info with
sumInvestmentBoxon box 2a. Bind0.line.13_capital_gain_distributionsto it and delete the todo. Boxes 2b and 2d (the 25% and 28% worksheets) stay later.Priority: next engine PR, with finding 1. Both change Schedule D line 16.
Test: one scenario with box 2a and at least one trade. Carter has 2a but no Schedule D, and jordan's 2a is zero, so no golden changes.
captaincole commentedon Nov 1, 2026
Finding 7: Form 8995-A line 34 ignores gains when there is no Schedule D
Our position: We agree the line is wrong, but the deduction is still right, so it can wait. Line 34 is qualified dividends plus net capital gain. With no Schedule D, net capital gain is 1040 line 7a (capital gain distributions only). Our binding sets it to zero in that case.
Lines 34 to 36 print wrong. Line 36 (20% of taxable income minus line 34) limits the deduction only when the QBI amount is bigger than it. That never happens for a filer at our target income, so the final deduction does not change.
Fix: in
1.line.34, when Schedule D line 16 exists, keep the current path. Otherwise use at least $0 of 1040 line 7a.Priority: later. It is a printed-form accuracy fix with no effect on the tax.
Test: a unit test with 1099-DIV box 2a and box 5 and no trades. Only jordan files 8995-A, and he has Schedule D, so no golden changes.
captaincole commentedon Nov 1, 2026
Finding 8: Form 8889 line 3 (last-month rule and the 55+ catch-up)
Our position: We agree with half of it.
decisions.hsa.last_month_rule), and we also need a fact for coverage on December 1.Fix: add an age-55 helper. In
0.line.3_annual_hsa_contribution_limit, add $1,000 × months / 12 for an unmarried 55+ filer. Fix the stale note on line 7. Filtering 1099-SA box 5 to "HSA" is a one-line cleanup.Priority: later. No scenario has an HSA filer over 55.
captaincole commentedon Nov 1, 2026
Finding 9: Form 8959 must-file uses $200,000 for every filing status
Our position: Partly right, and it can wait. The "Who Must File" section of the instructions makes any single W-2 with Medicare wages over $200,000 a must-file for every status. So the $200,000 test is a real rule, not a shortcut. It misses one case: married filing separately with wages between $125,000 and $200,000. Those filers owe the tax but do not get the form.
For MFJ, several W-2s that add up to $200,000 to $250,000 also trigger the form, with $0 tax. That is harmless.
Cade never sets
must_file_8959, so this fallback inresolve8959Infodecides every return.Fix: when the filing status is known, use its threshold ($125,000 MFS, $250,000 MFJ, $200,000 otherwise) next to the single-W-2 rule. When it is not known, keep $200,000.
Priority: later. MFS is out of scope.
Test: a unit test for MFS with $150,000 of wages.
Engine: capital loss limit and blank lines read as zero
captaincole commentedon Nov 1, 2026
Closing for now.
Part 2 (logic, line by line): done. The notes by form and the nine findings are in the comments above. #67 fixes seven of them. Findings 8 (Form 8889 catch-up and last-month rule) and 9 (Form 8959 MFS threshold) wait for scope. #68 adds a skill so the comparison can be repeated.
Part 1 (numbers): deferred. Our engine does not cover the full federal return yet, so a generated grid would mostly show known gaps. PolicyEngine also starts from computed amounts (taxable pension income, the HSA deduction, net gains), so it can check the back half of a return (AGI to total tax) but not the front half (documents to income lines). Reopen this, or start a new issue, when the engine's federal coverage is complete. A first step could be our own scenarios run through both engines.