From 9ca36fe46b7b6b2dae2f8069aa2cbee2cec9d33e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 05:16:23 -0400 Subject: [PATCH 01/22] Read the OBR's national living wage levels from economy Table 1.13 The importer finds the table by its title, reads the levels across its row of years and writes each year's rate over the year before's, to six decimals, then average_earnings year for year. A note that the rate has been set by the Government up until a year splits the rates set from the OBR's estimates in the series' comments. Any other layout is refused. Regenerating obr.national_living_wage from the October 2026 tables changes no value, only its comments and first reference. Co-Authored-By: Claude Opus 5.5 --- .../gov/economic_assumptions/yoy_growth.yaml | 36 +- .../efo_october_2026_economy_table_1_13.xlsx | Bin 0 -> 14466 bytes policyengine_uk/utils/import_obr_forecasts.py | 398 ++++++++++++++++++ 3 files changed, 416 insertions(+), 18 deletions(-) create mode 100644 policyengine_uk/tests/fixtures/obr_economy/efo_october_2026_economy_table_1_13.xlsx diff --git a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml index acef315895..2726ad906b 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml +++ b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml @@ -207,34 +207,34 @@ obr: national_living_wage: description: National living wage year-on-year growth, for the rate from April of each year. - # The OBR forecasts the national living wage itself (EFO detailed economy - # tables, Table 1.13): rates set to 2027, then "OBR estimates consistent - # with the rest of our economy forecast". Each value here is one year's - # rate over the year before's, to six decimals, so compounding them from - # £13.21 reproduces the table to the penny: £13.51 (2028), £13.85 (2029), - # £14.23 (2030) and £14.62 (2031). HM Treasury costed the Budget 2026 - # Carer's Allowance taper on these "OBR projections for increases in the - # National Living Wage" (policy costings, October 2026). After the EFO - # horizon the series takes average_earnings' long-run growth: the Low Pay - # Commission's reference point for the rate is two-thirds of median - # hourly earnings (remit 2026). Refresh 2028 onwards with each EFO. + # The OBR forecasts the national living wage itself, in £ an hour, in the + # EFO detailed economy tables (Table 1.13 in October 2026): the rates the + # Government has set, then "OBR estimates consistent with the rest of our + # economy forecast". utils/import_obr_forecasts.py reads the table and + # writes each year's rate over the year before's, to six decimals, so + # compounding them reproduces the table to the penny. Its comments below + # keep the years the Government has set apart from the OBR's estimates; + # gov.hmrc.minimum_wage cites the regulations or Budget behind each rate + # set. HM Treasury costed the Budget 2026 Carer's Allowance taper on these + # "OBR projections for increases in the National Living Wage" (policy + # costings, October 2026). After the EFO horizon the series takes + # average_earnings' long-run growth: the Low Pay Commission's reference + # point for the rate is two-thirds of median hourly earnings (remit 2026). values: - # Rates set: S.I. 2021/329 to 2026/357, and £13.21 for 2027 (Budget - # 2026, paragraph 6.128), from £8.72 in 2020. + # Set by the Government (2021-2027) from OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)": £8.72 in 2020 to £13.21 in 2027 2021-01-01: 0.021789 2022-01-01: 0.066218 2023-01-01: 0.096842 2024-01-01: 0.097889 2025-01-01: 0.067308 - 2026-01-01: 0.040950 + 2026-01-01: 0.04095 2027-01-01: 0.039339 - # OBR estimates (2028-2031), EFO October 2026 Table 1.13 + # OBR estimates (2028-2031) from OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)": £13.21 in 2027 to £14.62 in 2031 2028-01-01: 0.022984 2029-01-01: 0.025251 2030-01-01: 0.027289 2031-01-01: 0.026986 - # Long run (2032+): average_earnings growth, as refreshed by the - # October 2026 EFO (#2717) + # Beyond the OBR forecast: the average_earnings long-run path above, year for year 2032-01-01: 0.0285 2033-01-01: 0.0309 2034-01-01: 0.0334 @@ -281,7 +281,7 @@ obr: unit: /1 label: National living wage growth reference: - - title: OBR EFO October 2026, detailed forecast tables (economy), Table 1.13 + - title: OBR EFO October 2026 (detailed forecast tables, economy, Table 1.13) href: https://obr.uk/efo/economic-and-fiscal-outlook-october-2026/ - title: HM Treasury, Budget 2026 policy costings, Carer's Allowance taper (cost base) href: https://assets.publishing.service.gov.uk/media/6ae1883fe4367e5c8b0467df/AB26_Policy_Costing.pdf diff --git a/policyengine_uk/tests/fixtures/obr_economy/efo_october_2026_economy_table_1_13.xlsx b/policyengine_uk/tests/fixtures/obr_economy/efo_october_2026_economy_table_1_13.xlsx new file mode 100644 index 0000000000000000000000000000000000000000..df94de0c0b920cff0485cf34b584aca508e93cf4 GIT binary patch literal 14466 zcmc(`bCf5`vNqh^Gu=IHPTRI^+qP}}+O}=mwykMT+qP}Iv(H)UyKC=r?_b}Gs?1d@ zDkGn$ipt2S%8HPa{PrCg2nguMpNbC@@B*}8`3(qY@ec|2=Zvcrt&6RLrGc%jC5@}K zRgsdVZ2<@T2PN5782wf5qy#)4t3ronk-X!R;=Zq9sLys+Q) z&GSSWJEJFg&>=R(XMIIE>VevR(RD9Vj*-G5jsU?gGycKGa&w4|w{TRU{&-$Oc}EF_ zIC+W)tdn$;0{>bE2a&#(T;#wWrlxcyVQHeZy3AiKuvUY=dAE_Nc9KdT9Uzb7nzkAO zy|)9H1L+BV{V?)G92Ls!X6P%A*q@L!STU`PfUfER8(0lB@j;yAC^#WZb&Lq*Hc%s$ z=9gS~0x4ntp^-#j*o1bo-h*)0-eZyz=pZj}Ya|`(<;KajjAj$6V zykchvgY&w7mf|e)NL3=6*z!|BOxuT0_$*sCG4ls{k<_dckG-Bc_HmLQD`2J;3nkbD z1MlZ4ux!9nZ&IOzIb7>h>M}HD0O{kqmVyqVII1#v-f$NnPB}KOKL3juTH&PCMGFvf zKKLyzZT7m8Ho=!ed@8U^R#=3sic;kffY_^;d({WY-Hci0fQT3wRf$R7TqxB7H%VhE zUPB(fdnG5avH2Qh6^6rvXRLN;~PuT=W4m753iOV4WYcYK$_q4jfuAGY7lw0>`dC}j zW6itV^zQIW0xcv5t@_3q*oR0z&t{-(m&OHKR%?|^Vw)^8OxJi5xlMbuMzRWsI+QEG zoYc;~Ic_n46|dTr*@aj zI^L#?Tq9$miq;xy;ns@~q{OX3SeJl&wi_~5o?Nm6agTB0PhZXKS1OOrdx~bHm7}01 zTeLH31Eh~bg3wRn4utfP)vh*NTU8UlrD>NkiD)Z5b$)yC35|Q~gCcz5Qq->VG2;26 z8=8S}&CTQ0N-sbf4mjAghE}%1F`p@fg~UkPqVsSI237NDY*Jv=zc;KA+D)EFSKrJ@ z+eFwtoxNpnUFAPGBW)d?ZVmWuo?g_be8vFouh4P4Dz{31w4Z->VmH?H0harlT`JLx z&e#31M{*D#AdG+8rLKdqmE+&6ljg58UGd|v{&eudK1C;A9gK|>ogB<P9QN4audsiA`PxlbjqCpwR%jM> z?{FVz(LNCDZjWY_@F%j7FH8^7&Mpc22=YpA2|Hf;2>(p{jGz4O8YVWDI1ZSwksef1 zmFEKy+Yb^VxI6lKW_wu1h9hLX)~!Z6dxhNl#6xTUd_EYvn9MV&swH!N&)Wm#@oBs5 zElB!$J{ZCEj?Pcz*X4C@V+h+@d5l2)t=C^7(}*+C3fB|qZ+AL)J#F_+kRG&-E`l_* zSN7CD|4@ahIP##nj3@T~d=S3E%%ZBc+tEYpDeJwudRqB@>$v}U>-at%HTT5ExEN*a z+_+^3j;kM-he?WW>e1%}hIF?V+Vn^Qq{s>Qfb$&nhUqjKD_F?U9qf632ichzV8HcD z3dDbpdJu4_1(FaIUr~QCVg#s?_DCe6ax0V!AWI&pC$@_sdet0 zv|il9$7EwCPu&m(TRUn0`xtJ0j89G_ZwaWe#X9aaHnpB-~g52 znoEH?d{_>V{k-!xL=M!@5aOJ+Zcgdqa~K;c8vL(=`L2MXP+SXqV0-OzojTxi$sgLT z$Q`|Y1|7Q)@1$X-;uFJDl_((bs_=q7gFdS`>krQREw4jXJ;Cju0V?Z(jfH0px` zoG!lF;QUIm(qQmgen-*T?Q-_VLMoqJ{vvOD&| z7`PLC)_zZ*`)Yj$4zb*!4D$W0KgQRL$G86MHz^mpKWem4=aSV2jsorr##{UhX+lf{ zM*FLRUtH(ohV!f=8@$5c8$K8}IoMSpnpFdsLaGA`&~ zzDmfYXlmZs7I2Zl+UeC%Z^+bk%V3n&d9;(~kt@WmYTtBTY?+`QQ|>D;{1WnV$?6Z} zK)A?*R|7d*qVNiVc8J{(e81Hn0{%L}TXCt9<|XBmGQ~z$XK15{;6pbXX7IP@Vuc|V z4U{xZA^tQo#C}k`AMJ`C5OglQ_TUNDLFciyZphTjK~^0hPguv}xRi4|yF8+9j8m-I zz;XB7d{|p2i%U)7shBUP zi&xIhGbbQRr;CrlPZFhRGm6_RNf{&^7MmbeFtsoxs1>a&{Gp4B(grD{HP7HH&W|-j z)em^UIW^Ee;|!WnO2oK`3XqIZzzA8v7*WoT*but*cMzkKUtYK+r0-_aj4t^W+Sz(c zz^^UoKpg?5c%6y}*^SQP@|V9dcT9<;qgf-RHhXrm9y; zrZH?zy?$){4?A%Z?IBzw+IF%Sr7ms-&_Y-!IN(;wz_^(dz72#TM4ep5*LF(d(@;R8 zxULZGPHI>j-b@0rXzeh`*+VBPrcf-fpL5pr_Bo+zW7}^fwy?^`$fAXKPDCHx2wYZT zTvil}<(b_DZF^5hKTa2t7{0rinc5JoEB6wy?MpqdG>&^PK6xqKkcr*Y24ec%!&7xF zcV!K%VX$a-E+d4&zklVno%u^{4bC?3xOKg?n=FQl?m^_=o|l5Pa(dl*D$6VE3G(}h zpXmWk=%Zyebt(`ALIXTg_#U7WfM?V)UwMmaY+pcuqh%8Na+M87blpwyLu~hN-$40Q zNo8u5rnP?4oJ@Z@E8P|;{Q}pOB7W*llpqn4*PbOMJF3|9ND|M{hSQ=DzWkyvDJVJcjrDKGXIY3!x0cn^a@xxVC>@YfZ;skG+ zEOCxgQ)9-&=5zC92Pc6+C2kp2PjUVsg-iezY!Eucl@ZEe1|gYMBMx$f9T(WJHs0xI zf5YMxPAe}&)Sv7`^fIPc$)n+SN*PvC-^3KW}@1$7oIoqd}Zupx!BdsEY{*a3tk$e}UKr*kmh z%ub_$7&Dg3`zf;Uy1@(H;jTg-2e2D18yiZ}A&wyc-ln3fH*5NLFSU#gy4x8oQUW%x zlR&$BC*Y;%YYz3QAyO3R*RJENgI{<>AS#LdDQ;aMSwO)+2dWL$bh_HzO{9Ff+I)t) z$Z%V9>ZbvAR`nK6+tm%OEJgZK65nv+lb24>m2alpm^BZkha9sP`8(GAn=FoaTvQBy5AqjRV)Ix>nF?QuI_$$pp_A@ya_HH57{D)25=7T}dt zR?z94aY7wrix+rgx+TCG8~<*dyB(dNiXojZ<5%X3C~R(ZT#*upt8QiUX+QfCopK4s6J>Cd!xA4{G=~2}IsLhY4*ZiUlHCMZ)x*>omlE=U+dS^{)xTVf- z4@RzFpj=d+&iEi`z9!dX)+*h#j`ME4UyrO}+GHZvBHz02^XJc8zO^1V$7YIEsUsnm zrj3Q64ns5WZG|l2PKPpFel)qaG~gsiZi}W7&Wf|Bg)a8eWJ!I*j-E5j>Jl03MHSI9OTAYC&lc#|O zgqT(xY$N1)7O+v$2leh-?K7z!{ZiYt^_{t@QTi=Yboh5p))VVEy>Lva%{-e|CpbPy zZb(QW5|y1Fky95t7kot7igCkbWAqljdq z@h~i}fC?iF8tq!D!oWp-FFJo?C=m+yUo2>c65fc23!l*bIWCs-_c@6`wNGAZ%T+I@`e0qF4|}xK{M=Zm z#tQSjRzO%!Mh!hc`qiG#9Qk1Lac0uVHRlT}3E8u{pR~s~(A+)`fB_3y@-n4G{}@-< zpfw5c2P|;n^K>a&i8rBpa;Q`{(z!9z%iJV4{#%|cHQXK&UYyIuo zkPeNA5uWz9>`UUToIVjOkn4>YV{VG_PPm{f7NR&t7)A7R@UPSB;5vUS&WQU^itB*15wwH%jcgvzqI+z zidt@T)yi10pU@NADKe@YRKlb#Cq#uNLgdQiiSxd3P)l#sUW+ciQfeHP7gS=I(RuDF8mcEx(^CxQ$#?cc>vIx4+bj2c47}nXzjqKt1Mt!iFa=bk;Z3vhw`Ijdh zL$rZ&{z~%YPx7~kP=-gC*jbpW1M?}3i+~JYfanELf;J}0 zIuqV!E~6;|F6=SzS^v3vT;BkYn(6KmicZuX`Rc9a<00>E=*6|nZ7B(QO~#2xE-NvK zIvb)`GctWKFa8h&dFIIhjl)dnfmr?Ra7XXocjE=QR}hE1W40`A%n#`4`W%Paz74Ye ze6RU(Yn%*sea^W2WnUkkCm z(M0B4Xst7Zxq0Zu;PUwx zeKg=BEge`mBHpNMjUC81`_|~$WVz!l3$vRaK%oxP zl_*xOD?#)#F0AlV;y{sHD)yANt*;Qnl|euiLE+Xd=Ts$hkbaNnM@6l`FGncE{=j_s z^HQgHwR(~x?W@W_6FAjP~W$VcdOLrnrnC#0qF6cS3qNcC)LJzl{d&5D-s4` zI|zwN(3zJ%it;{e9Soe{ar_H^VM~G0=-WX5kMKiJy4hqk)!(|_7D2L@+-V1u@=UkN zS09rWkKTz(m}eGf8q`I+`mk41OVu0TZZ;c{m3 zXZM56l8xm90Cn}8koRT*^}#^Q$l5P>`a*(S(nw_j z&5Men%v5Ze_bC@`hn?9F)W}$2f zjP8PJ{L}K;Th3h^(M-1P1D>DeOy8q>)a-wn^zBhprgWi6Z6Q=IQkH}h41bJoVtQ}& z1||BGT8$34oHLX7@IRo#ndwDGdwQ&C7frg#*LARvAO^HsNzQOu6xY6MH4Pp@ORg z^CYX;PKfL_PJ8!F0Wh3nh*OJr6NgmLEZ~i zj+%=P$qaJq(*4QciLL6zz?3)E0p_2g559$aUdg-l+B2Kny<>$&#~V7=YB-e{ zjzvuqobS7_H}!TmdAD_wj14N$t+?Nf*xev9b+A|7`ONFCeM5bnU7B|bO{`SC_x(vO z`MbJPH#HCo;5W+^0tMsCWwF{Avq-@!Ct`|2yqDu4`QB>-TX*F|!6O$FTNT_h==6IdCJcj~TmF)C2z`-08sQq0R_`Ig}BIA2Ly# z9-aq}+MR?9Hu^OEW`Cx8ZuRT>(XKDy=DuEW{VR+Ilk)t#=6Xt{$%Swu3x=R$PuK{S z3@>!%4n?lhXHfZ3py4HVBJR&50|xqPQ~9xWvn~YX9YxX!c$dl@&Vi;?wTX68il3R^ zP-t+Z6OSU)Or-otH{(OSEmQ9p5b%TNy)9RcJiloI*)fB%01AMgdZ~8kq_Sx`P`)h~ zp0g8dBO}-~I9q6ATcp@dZPxfDx#ej;$UN#_FP1HE`fkxkKkwZm+Hb9TthseKKU+Mx z0+b={Yx+>ceSPcCPFHiGww>@;&@qZWA$es15MEbpdwnmWl$*sLN&^({h9W+lcry-A z_fEt-!(oKR;ifBU2XUemlV(5YK$p+3b@y;tF}6=JK3f~NW9BrM2|zt8BkEd0!E@aC zN>>o6CPf#wESn4PRPU(p1C=&i4a#^1E)j)|2UUyk0-ww01K@R&9N@2$$S4T^tgM9P zGwf1=ru^?DfR_lu$3knqYeoDWb(iRmJaO*^vHz3h5B~rC@+;q*L!C*QG@|&#=r>aq zReJH9@Z2at5Qlo99BBmdsZpFj7L`J2f-oeYUMx=rg?#?k|KCQ{mnfd`QH%6bdN3J8 zjTEjQX4Le^B8$y*ewEv>j(}gQ%>zOmuimEJG%JP{r=8Mv$DYz#| znzBuhDjHe;YUwIk8Kk(V{)?ubBvCP6!$@KhTw=~bj!c)4ppzyQjw(*57nut~40|mu zMj}Rhpq3_;AUf5SR#hQqY{g2 zfNn!gQbe5+Kpmj=N7nc`Dtq0*YiFuc76}n~;tCgs`4LD?$A_GVLn@qr*Z0OVWe^8$ zLYB~W9bqiJ>s5Cs-4|gnXlFbRIi^vHrWH=cU2E#mAmDmywS7NXkEauAoE=X^F zG-c1da~t=Y@cJxrfNlUF(@JOU6(0z&gLq#;P9YycoAbc>KJ+WA!z0t8YrF2PX+} za`KC$TC;q{IHAV*LvfGBaI7)=W3%jG(T8mivRCV~qpD$+G zYe;3=7Iwtx4#ej!>;oh1r|=Pan%0YM(oc?ai029uUfV!^WHaNfo624@@(oX#Q~D|^ zgs_{z$op_~wXDEU0&)_8LMYf-BK>I3CrE|$IAc0qL2rEv=l7Pupf13Gkp!_=b=Kw` zsgh1)$04J~rO@(2AoIu(Gvn(yd)8S=%ITI=@VwNuBoUOC!1_v$SY(>P1vD0}b)P~n zx*qMM*$3WOYn+3*%y|nX2GL|>+33*hCt7>AOz7&DZd@8`1QFOF1&~K-h+l(+C63ZmcK0;*5 zB=stqWJgIMNPOd&BICi0H!aw?Kh8tnSxVktdcUzSAhX=7o~)WHB^Pc)Anqr`=anB{ z5d)i;J!ORb>HEX2Cm;$-o%_*T1VYcaOgu(VVAjh}#CLRQ2oFaPW z+)?vQ%8niBD1=Q5a9CV7q6$0?l@Z_PcWx^%5BUR>pcrq6z~;?ml|Z-6=`76?Ns?o>%{7#px zRaooG*(G3_6a@K^)bnbbu!z6tDM9vOWvgYUFoIIMR$bTR9aE{(NBU2rqytvTUh9^D zm(^+fk>a|_#Ov@TiOLNEkvSqJlCT>(}6lcg8YEyWA6L8xE@Y^SkK2j%>p8<6bs+6V3X0Lw9CnaJ#&+02Mi z=R$FdJw|0suhd!Sp)k3ZuH=fHnWmP_c!KshfIs7;;#X7%+=|ty;z&hoe#En82WCn}QQ?#nF<;sS}^46a#l4NU4|!X6Y_LusWwvykuDP!CCwOHICp#d7vl0 zZs?&dd`^!7WDmY-i8S$T+&SJt+hx=K)f|ZVU$1*0hWdEl_U8yw;X?V4K&DR<0M5tJ z!UC%6;K=QIgY`xvHo`a03{r!NQo!G8i#1@n5?BT-;H>+u*l-RnXn%Vi(5MzG|8#D{ zB9W#~lXz8ys0BHGrZz6jukB50<*QP{ZADxF&kTx!G&I-Hlo#N0e{-XNJn!1#P9|IR z@nYAN(961zl;SKh^zT%I?bM3|K{kS{gnc6ODT`Dz7Rbh%vT|n9 zhDS3?$ZEW9wi_L|l57i#IIfPDv%7gI4oIs*or0YXObs#kK8N7WSH9?%fl4^8h5)2R z+MtdH8g&L)1YvMhzU+=4=ZQ-aekj5ZRy*!kD3{*Owu*D>{X_qaJA*~ivk}r}aw{R0g)~e~Ek3hE4;HRpN{;yj=Jzg(BrPn4ZNq6JpX1W+| zaDT0~;%;eh17$hDd?B|A1lhoFU&LGqbQuA6ImhSgipfHdq%&FVCkM;w6rwC;Gvot9 z3Bdc+Bn?uc^f(VHgH5d>GyrGVMa}%6zTXl!9$5u$7dCk|2z?h=6vc&K7xY>#`J98Y z*9p=A=n)qxLV822Gs7b5@lhpiea8QLum~t^C0ZPKr0fQpYVYwedG)t0hbG(KL_=bE zTCeZ;vA_b_{Vo9uT-Wo26IG4-3aKW8MGPJ(xY}Kz`Ohg|pnnh6unAy!Lxctb;^hSb zLi=|N!(TyEj%LQjPL6-V8U7sT|B7R%TC;S_SHl6je$s5(@_QJPv9(6$1{bnRkQ#z= zS)Mm?K4j)cTv4e7>fmf>+VX-EL;T>?JJ;zjX-~*;XXDYz5vcL`Y*v@l-R%^szb_uC z4(2A8DX>Z_oM!Fv`LMWe%2rC$Z--t|$Jn~p+0B2tyU6)+`d}XCMXg|?A*L75Sie^> zZp(rO6K)J=)wfTw-c#bbO+3h`MeaGA4lu|V*{S#bZNj}hkjJqcD89NDL-%VMozTNw zBHuMzxI;QK`Hlb96+7ABbBJOErKZ3--!XVC@a1Tu8P_BOo%BGG+e~&<28uiQmrsusqbyZ(HgJ`UQz=NV@aC9>B=Hl#|7rclq;m#YcC0cPP zP&d&6q$h6p*S1$2J1ZNJh_`y&dRCcSB@bM|9qx4gmS)($%ket&RJ}iW^rd z%3IfY53lQ#^NOFO(-jI-f@WP8(#f=Ut)YWcc**udq6-H3R@(DEk{Lu*Tm2;ODqn8^ zUY4F&Ccs3a#d*GG8H?SGUL)n0)f`ne^S6tT?Me7|>(Dy)q2PAU#~-(6L`v$4n?uAV|I*)lQ~Bn@bczbw8SCjquD)hxXbhfSCgeq6M9;eQJA zw-+s39U@r3Q>Y@IMF>+Wh*MCLqa#a2ruy^eFT`K6uV7p8M`N_uLxga-V4=HYAw`%{ z$$!UlA1$_WZIHGoh#j&zV_$9TrBsd+wp3F|LP>2&_rA` z@0Tm#L%{7C>&yKwZ$;a(j;0k}H-;ND7DrK%;YD{X+{B zj_(e}(R2Qc2iDrrzZt-5GFTnZ=(&CY{Y~n&wso{_^`!l(U2$jC1pVsaHsDSqnjn}1 z{O@8rPO*Blvi4jZE2}2yHIDwxHH>NG8Uk$_x5LfQDIVzW09lXL*nSljCl=Cr+r z{U6LJ`_``?fa4>e0ZBBJ3~DBX8sf!KSRr866@M{V|C6bC$b9V;k7Obnl-V1uBla%?u;?F?q{u% zizZ#D%2R3r56_4`w6NYgzbf4yR;B2H^b$gI?r&{dvqmu4wnTUI-qe{=`ORSbtpPIhy8Yr$dyyUDaBG zrJ6l^Q8=nC~PdoF>NKj9_W5jmyYW zq`peKTRR$9sat6Ga-}UlJl+ta+koVVk-#3sambV|_GC{Zyu(5+c%t}>CFx^&x}unx zqan(Kx(I+b-X+7BpGNxqES_%VjCt~8L9<|T_ZXVuiI5@t(c5;XnVF6|849RkBUxPM zE$ITZ;bU_)G@mox_?uJ%z~i3D>oBk9MLbp4icGGEMoMEz zWHy`%{)ppJc@3Wu^|GE2@v6(%I7#P780j<=u{)jNbet2mY4Fg*iXVGsAJ;9cdDsXX zaGSVNvMvr+X3Zw=thsJm&xwX<;>`$?^bzjXEEQbjfn&>a&RR>8F^Ox`I=BwX$6&Nx z#HIJ5Sd7I#>}#VSDX1TkGK6H@e{|~&vRw+xf*XQFEU~jU&Pb%4ZEo-PS{o_~XrnV^ zQc@N?n@(Kmtcg@wTvFD)#N69)V#(Q*9-=SF9^bB|vl^;X(i-3*HOH^oD{1$R3P~>y zZ;LUWVy;UIo4Eb>$Xpy5v(n_v53LFyZ|6{ItG+um*rBH!x0<`zrhpw%R`eR!-3I2F zt+;aJG2Sg*yxG>{Jqu}Fw=S|K+Z-3X747``;q?HznvqQst)ZSBI@t!{nsKn(niZNw znWLnC@L2yDI^+#2(p9^5X4Tec(fVQ8y7FnL@>Lb)l%F&hB9(w5pB#SYqLh=A9dBX2 z!0(b}oI^T0VAG*JvLI|m)|}kxeJc5Ium#Z4g4-NHdBLRyTt6;6t$qC?78)}d_DKm8 z2q+dF2nhXu_>N%yS6YVJip?GivKQ@)kKDryt0Y0$LKAmr%V3xxbUMv+q|llGk$mnx zQMSQnS4!f6@ICU0RIDpI7su1>wZr)KoJTf#jg%5OZ7@ka)|{U3L_*^-^d#?B>XX1+ zsf7j+(;g^hx`#?yjtK7;lk47Y(JF%qX0Du)prhuJ@m6z%>$ro4dZ!U+V`bZ@IgIYm zaJ59ASw7AfBPM6G?GY38gh(DOPk!Ih8Iy@3&Xt48d{n3&dKLbcMMko6Me+!4+<2Mm zN>W$q5wRFoiMqyBQmOgAy-yu^Ywk4q{Tp^06U!#(yTLP|HCm8m+$|GI=kD1m$bsOL z`x7B&y}75-?AsSU`o2GKO&L~i0`jK}^;Kz*tie9#(0f5@qvXAdIRn$zjVIzxuVB|q z#>V~n=s;J-iPO5E*`f5v&Uyy;T+N|bZXgfTxA3?sGg!lGB7}W@P3as`uD$+Jd|a~| zDG#PvIN3-ramXS3Jq-V% zWr<`7u)BVCQDq#-_9ISWJF{e@-LyMY2x`PK$B7fJeJy3&drE!F_Hm2jku7rt+Yl2o zP<}owL|2owXHK6E!xIxtL1;eZx`%|1L}hv<-oe`1Q}>Ami7pH`iV}JyS})Zr+bCxv zePoH9^38FF;b=>Yo9c2m@6c#ijwlcwvh4y8=jqwETuAdfyBLPg5Zq*Jo9st%SHvGR zCHP%7T5>e9=F+0y`nI6+tV$&)mdNHsV|^PFTv03raH(1HvS*)-Co*VN&q?Gw>*hP- z@_WO#)AH1T-#PkS^?Js*U^B%B>;uG#z*n#o`&AVL;toE%~Tf^Zy-^Le7!8ciFIOew#ZQ@ANxP3|7r7BnQ4ps{id zmm%1V5OrEN$o)>j0F5dkha657f-W45GkG4^?lXBsu-vCJ>K~C7(EFf$o+1TCaJ2#tlV5pvarZX{Rg^7-tHuu18uU}SA-ZlWIy#&z~A;!2e zQl`AvE+yLT^Hp(@AeBv7h;3G)(Fe}q$ISzL$!pZ{u-`N{B(%*Ua|c?l=t7Hca_PB* zJf-ZAUHL=BN(?2VaNKQ<5fFs}g>;t5>qlTWnhF`hLcev`AOj93%Z&??FA9K6P$4{# z<4ZeUJZ+9?_*eaRb<5ZyslGy3%yntI!EUvW@}Ju|UB)@8p8jXW!tAz7;ns$RF-l!F&U@@0RN7t-?gv^9TAwmWHz8BkCtB#&e-cw=COsgD$+3$ZU z(!IAw*8jtK+EhpI!n?p8YKfRcX(*dnk2Zc}gZb#81>=I`j8x@HqM&dlrQ{w8D^Q=m zY9AC-=9k>2R-xWZ-$Zy^nH=KL0svnnL}!YUIDOdJSS+Rcahu6now8pwoE7U zwg4&yV{O*J#dNN)L8T)4q_3qM*i&}*O(W%a dict[str, int]: + """A row's columns and years, if every non-empty cell holds a year.""" + cells = { + column: str(value).strip() + for column, value in row.items() + if str(value or "").strip() + } + if cells and all(YEAR_RE.match(text) for text in cells.values()): + return {column: int(text) for column, text in cells.items()} + return {} + + +def _set_until(rows: list[dict[str, str | None]], where: str) -> int: + """The last year the Government has set, from the table's note.""" + notes = [ + normalise_label(value) + for row in rows + for value in row.values() + if SET_BY_GOVERNMENT_RE.search(normalise_label(value)) + ] + if len(notes) != 1: + raise ValueError( + f"{where} needs one note saying which years the Government has set " + f"and which are OBR estimates; found {len(notes)}" + ) + set_until = SET_BY_GOVERNMENT_RE.search(notes[0]).group(1) + estimates = OBR_ESTIMATES_RE.search(notes[0]) + if estimates is None or estimates.group(1) != set_until: + raise ValueError( + f"{where}'s note does not say the figures beyond {set_until} are " + f"OBR estimates: {notes[0]!r}" + ) + return int(set_until) + + +def find_level_line( + sheets: dict[str, list[dict[str, str | None]]], spec: LevelSeriesSpec +) -> LevelLine: + """The levels in the one row labelled ``spec.row`` of the one table + titled ``spec.title``, read across its row of years. + + Any other layout is refused rather than read: another unit, a second + row of years or of the label, a level missing under a year or one with + no year above it, or years that skip or repeat. + """ + name = ".".join(spec.path) + tables = [ + (table, sheet, table_rows) + for sheet, rows in sheets.items() + for table, title, table_rows in sheet_tables(rows) + if title == spec.title + ] + if len(tables) != 1: + found = ", ".join(f"sheet {sheet}" for _, sheet, _ in tables) or "none" + raise ValueError(f'Expected one "{spec.title}" table for {name}; found {found}') + table, sheet, rows = tables[0] + where = f"Table {table} (sheet {sheet})" + year_rows = [index for index, row in enumerate(rows) if _year_columns(row)] + if len(year_rows) != 1: + raise ValueError( + f"{where} needs one row of years across its columns; found {len(year_rows)}" + ) + header = year_rows[0] + years = _year_columns(rows[header]) + units = [ + normalise_label(value) + for row in rows[:header] + for value in row.values() + if normalise_label(value) + ] + if units != [spec.unit]: + raise ValueError( + f"{where} must declare its unit as {spec.unit!r} above its years; " + f"found {units}" + ) + ordered = [years[column] for column in sorted(years, key=column_index)] + if ordered != list(range(ordered[0], ordered[0] + len(ordered))): + raise ValueError(f"{where}'s years skip, repeat or run backwards: {ordered}") + first_year_column = min(column_index(column) for column in years) + matches = [] + for index, row in enumerate(rows): + labels = [ + value + for column, value in sorted( + row.items(), key=lambda item: column_index(item[0]) + ) + if column_index(column) < first_year_column and normalise_label(value) + ] + if labels and receipts_label(labels[0]) == spec.row: + matches.append((index, labels[0], row)) + if len(matches) != 1: + raise ValueError( + f'Expected one "{spec.row}" row in {where} for {name}; found {len(matches)}' + ) + index, label, row = matches[0] + if index < header: + raise ValueError(f'{where} has its "{spec.row}" row above its years') + stray = sorted( + column + for column, value in row.items() + if column_index(column) >= first_year_column + and column not in years + and str(value or "").strip() + ) + if stray: + raise ValueError(f"{where} has {name} values with no year above them: {stray}") + levels = {} + for column, year in years.items(): + text = row.get(column) + try: + level = float(text) + except (TypeError, ValueError): + level = None + if level is None or not math.isfinite(level) or level <= 0: + raise ValueError(f"{where} has no positive {year} {name} level: {text!r}") + levels[year] = level + set_until = _set_until(rows, where) + if not min(levels) <= set_until <= max(levels): + raise ValueError( + f"{where} says the Government has set the rate up until " + f"{set_until}, outside its years {min(levels)}-{max(levels)}" + ) + if TABLE_NUMBER_RE.fullmatch(sheet): + table = sheet + return LevelLine( + table=table, + label=FOOTNOTE_RE.sub("", " ".join(str(label).split())), + levels=dict(sorted(levels.items())), + set_until=set_until, + ) + + +def compute_level_growth( + xlsx_bytes: bytes, specs: tuple[LevelSeriesSpec, ...] = LEVEL_SERIES_SPECS +) -> dict[tuple[str, ...], LevelGrowth]: + """Growth in each level series from an EFO's economy tables.""" + sheets = read_workbook_sheets(xlsx_bytes) + result = {} + for spec in specs: + name = ".".join(spec.path) + line = find_level_line(sheets, spec) + values = { + year: round(line.levels[year] / line.levels[year - 1] - 1, 6) + 0.0 + for year in sorted(line.levels)[1:] + } + # As for the receipts series: the model compounds the growth into an + # index rounded to five decimals, which must stay above zero. + index = 1.0 + for year, value in values.items(): + index = round(index * (1 + value), 5) + if not (math.isfinite(index) and index > 0): + raise ValueError( + f"{name}: {year} growth of {value} takes the index to " + f"{index} at five decimals, which the model cannot " + "compound from" + ) + result[spec.path] = LevelGrowth(values, line) + return result + + +def average_earnings_years(content: str, after_year: int) -> dict[int, float]: + """``average_earnings`` after ``after_year``, year for year: the path + ``average_earnings_path`` gives, with an entry in every year + average_earnings has one, so the two series list the same years.""" + values = yaml.safe_load(content)["obr"]["average_earnings"]["values"] + later = { + int(str(instant)[:4]): float(value) + for instant, value in values.items() + if int(str(instant)[:4]) > after_year + } + return dict(sorted({**average_earnings_path(content, after_year), **later}.items())) + + +def render_level_values( + old_lines: list[str], + indent: int, + growth: LevelGrowth, + tail: dict[int, float], + release: tuple[str, int], +) -> list[str]: + """A level series' values entries: earlier years as they were, then + the growth into the rates the Government has set, then into the OBR's + estimates, then the average_earnings path.""" + first_year = min(growth.values) + kept: list[str] = [] + pending: list[str] = [] + for line in old_lines: + match = VALUE_LINE_RE.match(line) + if match is None: + if line.strip() and not line.strip().startswith("#"): + raise ValueError(f"Unexpected line in a values block: {line!r}") + pending.append(line) + continue + if int(match.group(1)) < first_year: + kept += pending + [line] + pending = [] + + month, year = release + published = growth.line + pad = " " * indent + lines = list(kept) + source = ( + f'OBR EFO {month} {year}, economy Table {published.table} "{published.label}"' + ) + for kind, years in ( + ( + "Set by the Government", + [y for y in growth.values if y <= published.set_until], + ), + ("OBR estimates", [y for y in growth.values if y > published.set_until]), + ): + if not years: + continue + span = f"{years[0]}" if len(years) == 1 else f"{years[0]}-{years[-1]}" + start, end = years[0] - 1, years[-1] + lines.append( + f"{pad}# {kind} ({span}) from {source}: " + f"£{published.levels[start]:.2f} in {start} to " + f"£{published.levels[end]:.2f} in {end}" + ) + lines += [f"{pad}{y}-01-01: {format_growth(growth.values[y])}" for y in years] + lines.append( + f"{pad}# Beyond the OBR forecast: the average_earnings long-run path " + "above, year for year" + ) + lines += [f"{pad}{y}-01-01: {format_growth(v)}" for y, v in tail.items()] + return lines + + +def render_level_series_yaml( + content: str, + growth: dict[tuple[str, ...], LevelGrowth], + release: tuple[str, int], +) -> str: + """Rewrite each level series' values and first reference in a + yoy_growth.yaml. + + Years before the table's second year are kept, so the series' index + keeps its base year, and its other references stay. Years after the + table follow average_earnings as ``content`` holds it. The result is + checked to differ from ``content`` only in those values and that + reference. + """ + month, year = release + expected = copy.deepcopy(yaml.safe_load(content)) + lines = content.split("\n") + for path, series in growth.items(): + tail = average_earnings_years(content, max(series.values)) + start, end = _series_block(lines, ("obr", *path)) + indent = 2 * len(path) + 2 + values_start, values_end = _yaml_block(lines, "values", indent, start, end) + old_values = lines[values_start + 1 : values_end] + entry_indent = next( + (_indent(line) for line in old_values if VALUE_LINE_RE.match(line)), + indent + 2, + ) + new_values = render_level_values( + old_values, entry_indent, series, tail, release + ) + lines = [*lines[: values_start + 1], *new_values, *lines[values_end:]] + + start, end = _series_block(lines, ("obr", *path)) + metadata = _yaml_block(lines, "metadata", indent, start, end) + ref_start, ref_end = _yaml_block( + lines, "reference", indent + 2, metadata[0], metadata[1] + ) + pad = " " * (indent + 2) + title = next( + ( + index + for index in range(ref_start + 1, ref_end) + if lines[index].startswith(f"{pad} - title: ") + ), + None, + ) + if title is None or not lines[title + 1].startswith(f"{pad} href: "): + raise ValueError(f"Could not find the first reference of {'.'.join(path)}") + reference = { + "title": build_reference_title(month, year, series.line.table), + "href": build_efo_href(month, year), + } + lines[title] = f"{pad} - title: {reference['title']}" + lines[title + 1] = f"{pad} href: {reference['href']}" + + node = expected["obr"] + for key in path: + node = node[key] + kept = { + instant: value + for instant, value in node["values"].items() + if int(str(instant)[:4]) < min(series.values) + } + node["values"] = { + **kept, + **{ + date(entry_year, 1, 1): value + for entry_year, value in {**series.values, **tail}.items() + }, + } + node["metadata"]["reference"][0] = reference + + rendered = "\n".join(lines) + if yaml.safe_load(rendered) != expected: + raise ValueError( + "Rewriting the level series changed more than their values and " + "first reference" + ) + return rendered + + def infer_workbook_release(xlsx_bytes: bytes) -> tuple[str, int] | None: """The release a detailed forecast tables workbook names on its contents sheet, e.g. "March 2026 Economic and fiscal outlook - detailed forecast @@ -1752,6 +2117,19 @@ def print_receipts_summary(growth: dict[tuple[str, ...], ReceiptsGrowth]) -> Non print(f"- {'.'.join(path)} (Table {series.line.table}): {window}") +def print_level_summary(growth: dict[tuple[str, ...], LevelGrowth]) -> None: + print(f"Updating {len(growth)} OBR level series") + for path, series in growth.items(): + window = ", ".join( + f"{year}: {format_growth(value)}" + for year, value in sorted(series.values.items()) + ) + print( + f"- {'.'.join(path)} (Table {series.line.table}, set by the " + f"Government up until {series.line.set_until}): {window}" + ) + + def build_arg_parser() -> argparse.ArgumentParser: parser = argparse.ArgumentParser( description="Import OBR economy forecast tables into yoy_growth.yaml.", @@ -1819,6 +2197,15 @@ def build_arg_parser() -> argparse.ArgumentParser: "duty inputs" ), ) + parser.add_argument( + "--skip-national-living-wage", + action="store_true", + help=( + "Do not update national_living_wage, which projects the minimum " + "wage rates, from the economy tables' national living wage table " + "(Table 1.13 in October 2026)" + ), + ) parser.add_argument( "--gaps-only", action="store_true", @@ -2006,6 +2393,12 @@ def main(argv: list[str] | None = None) -> int: ) receipts_growth = compute_receipts_growth(receipts_bytes, previous_bytes) print_receipts_summary(receipts_growth) + # The economy tables hold the national living wage's levels, so it moves + # with the economy series unless skipped. + level_growth = None + if not args.gaps_only and not args.skip_national_living_wage: + level_growth = compute_level_growth(workbook_bytes) + print_level_summary(level_growth) calendar_values = ( read_calendar_values(source_yaml_path) if args.gaps_only else series_values ) @@ -2045,6 +2438,11 @@ def main(argv: list[str] | None = None) -> int: outputs[yoy_path] = render_receipts_growth_yaml( outputs[yoy_path], receipts_growth, (month, year), previous_release ) + if level_growth is not None: + yoy_path = args.output or source_yaml_path + outputs[yoy_path] = render_level_series_yaml( + outputs[yoy_path], level_growth, (month, year) + ) if write_gaps: gap_dir = get_forecast_gap_dir(source_yaml_path) for spec in STATUTORY_GAP_SPECS: From 13b2029b15c9846297128c22640ba7dd8da9ae09 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 05:46:47 -0400 Subject: [PATCH 02/22] Test the national living wage import from Table 1.13 The tests read the real October 2026 table and check that the committed series is what the importer writes from it. They refuse each layout change that could put another row, unit or year into the series, and check the importer's main run. Hypothesis properties check the model's index against the published levels and that it stays above zero. The projection tests check that their copy of the table is the fixture's, and that the index carries any year of it to any later year to the penny. The importer now checks where the labelled row sits before its unit, so a row above the years is named as such. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 552 +++++++++++++++++- .../tests/test_minimum_wage_projection.py | 37 +- policyengine_uk/utils/import_obr_forecasts.py | 28 +- 3 files changed, 601 insertions(+), 16 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index e778774825..6145da288c 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -1,3 +1,4 @@ +import math import os from dataclasses import replace from datetime import date @@ -10,7 +11,11 @@ import yaml from hypothesis import event, given, settings from hypothesis import strategies as st +from policyengine_core.parameters import ParameterNode +from policyengine_uk.parameters.gov.economic_assumptions.create_economic_assumption_indices import ( + create_economic_assumption_indices, +) from policyengine_uk.utils import import_obr_forecasts from policyengine_uk.utils.import_obr_forecasts import ( @@ -23,6 +28,8 @@ ReceiptsSeriesSpec, machine_games_spec, average_earnings_path, + average_earnings_years, + compute_level_growth, compute_receipts_growth, find_receipts_line, format_gap, @@ -33,6 +40,7 @@ read_calendar_values, read_workbook_sheets, render_forecast_gap_yaml, + render_level_series_yaml, render_receipts_growth_yaml, render_yoy_growth_yaml, build_efo_href, @@ -75,6 +83,7 @@ def make_test_xlsx(earnings_quarter: bool = True) -> bytes: + """ @@ -83,6 +92,7 @@ def make_test_xlsx(earnings_quarter: bool = True) -> bytes: + """ @@ -176,6 +186,7 @@ def make_test_xlsx(earnings_quarter: bool = True) -> bytes: archive.writestr("xl/worksheets/sheet1.xml", sheet_16) archive.writestr("xl/worksheets/sheet2.xml", sheet_17) archive.writestr("xl/worksheets/sheet3.xml", sheet_116) + archive.writestr("xl/worksheets/sheet4.xml", nlw_sheet(nlw_rows())) return buffer.getvalue() @@ -812,9 +823,27 @@ def test_small_gaps_are_written_in_fixed_point(value, text): ) + "\n" ) +LEVEL_SERIES_FIXTURE = """ national_living_wage: + description: National living wage growth. + # The series' own comment stays. + values: + # From an older table + 2021-01-01: 0.0100 + # Older estimates + 2024-01-01: 0.0100 + 2031-01-01: 0.0200 + metadata: + unit: /1 + reference: + - title: Old + href: https://example.com/old + - title: Kept + href: https://example.com/kept +""" YOY_GROWTH_FIXTURE = ( "obr:\n" + "".join(FLAT_SERIES_FIXTURE.format(key=key) for key in SERIES_KEYS[:2]) + + LEVEL_SERIES_FIXTURE + RECEIPTS_FIXTURE + "".join(FLAT_SERIES_FIXTURE.format(key=key) for key in SERIES_KEYS[2:]) + """ons: @@ -1442,7 +1471,7 @@ def test_read_calendar_values_includes_nested_series(tmp_path): 2025: 0.01, 2031: 0.02, } - assert set(values) == set(SERIES_KEYS) | { + assert set(values) == set(SERIES_KEYS) | {"national_living_wage"} | { ".".join(path) for path in ALL_RECEIPTS_PATHS.values() } @@ -2504,3 +2533,524 @@ def test_a_written_series_keeps_every_index_positive(levels, schedules): for year in sorted(series.values): index = round(index * (1 + series.values[year]), 5) assert index > 0 + + +# The national living wage (#2721): economy Table 1.13 publishes its levels, +# £ an hour, across a row of calendar years, and the importer writes their +# growth. + +NLW_PATH = ("national_living_wage",) +NLW_LEVELS = (10.0, 10.5, 11.0, 11.55, 12.0) +NLW_NOTE = ( + "Note: The NLW has been set by the Government up until {set_until} " + "following recommendations from the Low Pay Commission (LPC) in accordance " + "with its remit, and the state of the labour market and economy. Figures " + "beyond {estimates_after} are OBR estimates consistent with the rest of our " + "economy forecast. " +) + + +def nlw_rows(levels=NLW_LEVELS, first_year=2023, set_until=2025): + """Table 1.13's cells by row and column, laid out as the OBR's: the + title in B2, the unit in C3, years across row 4 from column C, the + levels in row 5 (labelled in B5) and the note in B6.""" + columns = [column_letter(3 + i) for i in range(len(levels))] + return { + 1: {"A": "Back to contents"}, + 2: {"B": "1.13 National Living Wage "}, + 3: {"C": "£ per hour "}, + 4: {column: first_year + i for i, column in enumerate(columns)}, + 5: {"B": "National Living Wage (NLW)", **dict(zip(columns, levels))}, + 6: {"B": NLW_NOTE.format(set_until=set_until, estimates_after=set_until)}, + } + + +def nlw_sheet(rows) -> bytes: + """A sheet holding ``rows``: text as inline strings, numbers as numbers.""" + return make_sheet( + { + number: [ + make_inline_cell(f"{column}{number}", value) + if isinstance(value, str) + else make_number_cell(f"{column}{number}", value) + for column, value in sorted( + cells.items(), + key=lambda item: import_obr_forecasts.column_index(item[0]), + ) + ] + for number, cells in rows.items() + } + ) + + +def nlw_workbook(rows=None) -> bytes: + return make_workbook({"1.13": nlw_sheet(nlw_rows() if rows is None else rows)}) + + +def model_index(values): + """The index the model compounds from a growth series + (create_economic_assumption_indices on a tree holding only it).""" + tree = ParameterNode( + "", + data={ + "gov": { + "economic_assumptions": { + "yoy_growth": { + "obr": { + "national_living_wage": { + "values": { + f"{year}-01-01": value + for year, value in values.items() + } + } + } + } + } + } + }, + ) + create_economic_assumption_indices(tree) + index = tree.gov.economic_assumptions.indices.obr.national_living_wage + return {year: index(f"{year}-01-01") for year in range(min(values), 2040)} + + +def test_level_growth_is_the_ratio_of_consecutive_levels(): + growth = compute_level_growth(nlw_workbook())[NLW_PATH] + assert growth.line.levels == dict(zip(range(2023, 2028), NLW_LEVELS)) + assert growth.line.set_until == 2025 + assert growth.line.table == "1.13" + assert growth.line.label == "National Living Wage (NLW)" + # 10.5 / 10, 11 / 10.5, 11.55 / 11 and 12 / 11.55, to six decimals. + assert growth.values == {2024: 0.05, 2025: 0.047619, 2026: 0.05, 2027: 0.038961} + + +def test_years_written_as_text_read_as_years(): + rows = nlw_rows() + rows[4] = {column: str(year) for column, year in rows[4].items()} + assert compute_level_growth(nlw_workbook(rows)) == compute_level_growth( + nlw_workbook() + ) + + +def test_levels_laid_out_down_a_column_are_refused(): + rows = { + 2: {"B": "1.13 National Living Wage"}, + 3: {"C": "£ per hour"}, + **{4 + i: {"B": 2023 + i, "C": level} for i, level in enumerate(NLW_LEVELS)}, + 10: {"B": NLW_NOTE.format(set_until=2025, estimates_after=2025)}, + } + with pytest.raises(ValueError, match="one row of years across its columns"): + compute_level_growth(nlw_workbook(rows)) + + +def test_the_table_must_appear_once(): + sheet = nlw_sheet(nlw_rows()) + with pytest.raises(ValueError, match="found sheet 1.13, sheet 1.13b"): + compute_level_growth(make_workbook({"1.13": sheet, "1.13b": sheet})) + + +REAL_TABLE_1_13 = ( + Path(__file__).parent + / "fixtures" + / "obr_economy" + / "efo_october_2026_economy_table_1_13.xlsx" +).read_bytes() +# Table 1.13 of the October 2026 economy tables, £ an hour, as its cells +# hold it. +OCTOBER_2026_NLW = { + 2020: "8.7200000000000006", 2021: "8.91", 2022: "9.5", 2023: "10.42", + 2024: "11.44", 2025: "12.21", 2026: "12.71", 2027: "13.21", + 2028: "13.513623297578135", 2029: "13.854853379339588", + 2030: "14.23293930734993", 2031: "14.617023376183901", +} # fmt: skip +# The series as maintainers kept it by hand before the importer read the +# table (#2700): every year is its level over the year before's. +HAND_KEPT_NLW_GROWTH = { + 2021: 0.021789, 2022: 0.066218, 2023: 0.096842, 2024: 0.097889, + 2025: 0.067308, 2026: 0.040950, 2027: 0.039339, 2028: 0.022984, + 2029: 0.025251, 2030: 0.027289, 2031: 0.026986, +} # fmt: skip + + +def test_the_october_2026_table_is_read_across_its_row(): + """The real Table 1.13: the level under each year, the rates set by the + Government up until 2027, and the growth the hand-kept series held.""" + growth = compute_level_growth(REAL_TABLE_1_13)[NLW_PATH] + assert growth.line.levels == { + year: float(text) for year, text in OCTOBER_2026_NLW.items() + } + assert growth.line.set_until == 2027 + assert growth.line.table == "1.13" + assert growth.values == HAND_KEPT_NLW_GROWTH + assert infer_workbook_release(REAL_TABLE_1_13) == ("October", 2026) + + +def test_october_2026_table_1_13_reproduces_the_committed_series(): + """yoy_growth.yaml holds exactly what the importer writes from the real + Table 1.13: rerunning it on the committed file changes nothing.""" + committed = get_yoy_growth_path().read_bytes() + rendered = render_level_series_yaml( + committed.decode(), compute_level_growth(REAL_TABLE_1_13), ("October", 2026) + ) + assert rendered.encode() == committed + + +def edit_real_table_1_13(edits): + """The real October 2026 workbook with cells of sheet 1.13 set: text as + an inline string, a number as a number, None to empty the cell. Every + other archive entry is unchanged.""" + output = BytesIO() + with ZipFile(BytesIO(REAL_TABLE_1_13)) as source, ZipFile(output, "w") as dest: + path = _sheet_paths(source)["1.13"] + sheet = ET.fromstring(source.read(path)) + data = sheet.find("m:sheetData", XLSX_NS) + for ref, value in edits.items(): + number = ref.lstrip("ABCDEFGHIJKLMNOPQRSTUVWXYZ") + row = data.find(f"m:row[@r='{number}']", XLSX_NS) + if row is None: + row = ET.SubElement(data, f"{{{XLSX_MAIN}}}row", r=number) + cell = row.find(f"m:c[@r='{ref}']", XLSX_NS) + if cell is not None: + row.remove(cell) + if value is None: + continue + cell = ET.SubElement(row, f"{{{XLSX_MAIN}}}c", r=ref) + if isinstance(value, str): + cell.set("t", "inlineStr") + text = ET.SubElement(cell, f"{{{XLSX_MAIN}}}is") + ET.SubElement(text, f"{{{XLSX_MAIN}}}t").text = value + else: + ET.SubElement(cell, f"{{{XLSX_MAIN}}}v").text = repr(value) + for entry in source.infolist(): + dest.writestr( + entry, + ET.tostring(sheet) if entry.filename == path else source.read(entry), + ) + return output.getvalue() + + +NLW_COLUMNS = "CDEFGHIJKLMN" +REAL_LAYOUT_CHANGES = { + "retitled": ( + {"B2": "1.13 National Minimum Wage"}, + 'Expected one "national living wage" table for national_living_wage; ' + "found none", + ), + "unit_in_pence": ( + {"C3": "pence per hour"}, + "must declare its unit as '£ per hour'", + ), + "second_unit": ({"D3": "per cent"}, "must declare its unit"), + "no_unit": ({"C3": None}, "must declare its unit"), + "row_relabelled": ( + {"B5": "National Living Wage"}, + r'Expected one "national living wage \(nlw\)" row .* found 0', + ), + "second_row": ( + {"B7": "National Living Wage (NLW)", "C7": 9.0}, + r'Expected one "national living wage \(nlw\)" row .* found 2', + ), + "row_above_years": ( + {"B3": "National Living Wage (NLW)", "B5": None}, + "row above its years", + ), + "level_missing": ({"N5": None}, "no positive 2031 national_living_wage level"), + "level_as_text": ({"K5": "n/a"}, "no positive 2028 national_living_wage level"), + "level_of_zero": ({"C5": 0.0}, "no positive 2020 national_living_wage level"), + "level_below_zero": ({"D5": -8.91}, "no positive 2021 national_living_wage level"), + "level_without_a_year": ({"O5": 15.0}, r"no year above them: \['O'\]"), + "year_skipped": ({"N4": 2032}, "years skip, repeat or run backwards"), + "year_repeated": ({"N4": 2030}, "years skip, repeat or run backwards"), + "years_backwards": ({"C4": 2031, "N4": 2020}, "years skip, repeat or run"), + "second_row_of_years": ( + {"C8": 2020, "D8": 2021}, + "one row of years across its columns; found 2", + ), + "labelled_row_of_years": ( + {"B4": "Calendar year"}, + "one row of years across its columns; found 0", + ), + "no_note": ({"B6": None}, "needs one note .* found 0"), + "second_note": ( + {"B9": NLW_NOTE.format(set_until=2027, estimates_after=2027)}, + "needs one note .* found 2", + ), + "note_disagrees": ( + {"B6": NLW_NOTE.format(set_until=2027, estimates_after=2028)}, + "does not say the figures beyond 2027 are OBR estimates", + ), + "set_beyond_the_table": ( + {"B6": NLW_NOTE.format(set_until=2035, estimates_after=2035)}, + "up until 2035, outside its years 2020-2031", + ), +} + + +@pytest.mark.parametrize( + "edits, message", REAL_LAYOUT_CHANGES.values(), ids=REAL_LAYOUT_CHANGES.keys() +) +def test_a_changed_layout_of_the_real_table_is_refused(edits, message): + """Each change to the real Table 1.13 that could put another row, unit + or year's figure into the series stops the importer instead.""" + with pytest.raises(ValueError, match=message): + compute_level_growth(edit_real_table_1_13(edits)) + + +def test_the_real_table_reads_the_same_with_no_cell_changed(): + assert compute_level_growth(edit_real_table_1_13({})) == compute_level_growth( + REAL_TABLE_1_13 + ) + + +def test_the_labelled_row_is_read_not_the_row_below_the_years(): + """With another rate's row between the years and the national living + wage, the levels still come from the row labelled for it.""" + edits = { + "B5": "National Minimum Wage (18-20)", + **{f"{column}5": 1.0 for column in NLW_COLUMNS}, + "B7": "National Living Wage (NLW)", + **{ + f"{column}7": float(OCTOBER_2026_NLW[2020 + i]) + for i, column in enumerate(NLW_COLUMNS) + }, + } + assert ( + compute_level_growth(edit_real_table_1_13(edits))[NLW_PATH].values + == HAND_KEPT_NLW_GROWTH + ) + + +NLW_BLOCK = """ national_living_wage: + description: National living wage growth. + # The series' own comment stays. + values: + # From an older table + 2021-01-01: 0.0100 + # Set by the Government (2024-2025) from OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)": £10.00 in 2023 to £11.00 in 2025 + 2024-01-01: 0.0500 + 2025-01-01: 0.047619 + # OBR estimates (2026-2027) from OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)": £11.00 in 2025 to £12.00 in 2027 + 2026-01-01: 0.0500 + 2027-01-01: 0.038961 + # Beyond the OBR forecast: the average_earnings long-run path above, year for year + 2028-01-01: 0.0000 + 2031-01-01: 0.0200 + metadata: + unit: /1 + reference: + - title: OBR EFO March 2026 (detailed forecast tables, economy, Table 1.13) + href: https://obr.uk/efo/economic-and-fiscal-outlook-march-2026/ + - title: Kept + href: https://example.com/kept +""" + + +def render_nlw(content=YOY_GROWTH_FIXTURE, workbook=None, release=("March", 2026)): + growth = compute_level_growth(workbook or nlw_workbook()) + return render_level_series_yaml(content, growth, release) + + +def test_render_writes_set_rates_estimates_then_average_earnings(): + """Years before the table's second year stay; the rates the Government + set and the OBR's estimates each get a comment naming their levels; then + average_earnings as the fixture holds it after 2027 (0% from 2026, 2% + from 2031), year for year. Only the first reference moves to the + release.""" + rendered = render_nlw() + assert LEVEL_SERIES_FIXTURE in YOY_GROWTH_FIXTURE + assert rendered == YOY_GROWTH_FIXTURE.replace(LEVEL_SERIES_FIXTURE, NLW_BLOCK) + assert render_nlw(rendered) == rendered + + +def test_the_series_lists_every_average_earnings_year_after_the_table(): + content = YOY_GROWTH_FIXTURE.replace( + " 2031-01-01: 0.0200\n", + " 2031-01-01: 0.0200\n 2032-01-01: 0.0200\n 2033-01-01: 0.0300\n", + ) + values = _values(render_nlw(content), NLW_PATH) + earnings = _values(content, ("average_earnings",)) + assert {y for y in values if y > 2027} == {2028} | {y for y in earnings if y > 2027} + for year in range(2028, 2050): + assert _in_effect(values, year) == _in_effect(earnings, year) + + +def test_average_earnings_years_starts_with_the_value_in_effect(): + content = YOY_GROWTH_FIXTURE + assert average_earnings_years(content, 2027) == {2028: 0.0, 2031: 0.02} + assert average_earnings_years(content, 2030) == {2031: 0.02} + assert average_earnings_years(content, 2031) == {2032: 0.02} + + +def test_a_series_without_a_first_reference_is_refused(): + content = YOY_GROWTH_FIXTURE.replace( + LEVEL_SERIES_FIXTURE, + LEVEL_SERIES_FIXTURE.split(" reference:\n")[0] + " reference: []\n", + ) + with pytest.raises(ValueError, match="first reference of national_living_wage"): + render_nlw(content) + + +def test_main_writes_the_national_living_wage_from_the_economy_tables(tmp_path, capsys): + yoy, _ = write_tree(tmp_path) + assert run_main(tmp_path, make_test_xlsx(), "--yaml-path", str(yoy)) == 0 + # 2028 takes the 2026 average earnings this run writes (3.33%). + assert _values(yoy.read_text(), NLW_PATH) == { + 2021: 0.01, + 2024: 0.05, + 2025: 0.047619, + 2026: 0.05, + 2027: 0.038961, + 2028: 0.0333, + 2031: 0.02, + } + assert "set by the Government up until 2025" in capsys.readouterr().out + + +@pytest.mark.parametrize( + "workbook, message", + [ + ( + rebuild_workbook(make_test_xlsx(), drop={"1.13"}), + 'Expected one "national living wage" table', + ), + ( + rebuild_workbook( + make_test_xlsx(), + drop={"1.13"}, + first={"1.13": nlw_sheet({**nlw_rows(), 3: {"C": "pence per hour"}})}, + ), + "must declare its unit as '£ per hour'", + ), + ], + ids=["no_table", "unit_in_pence"], +) +def test_main_writes_nothing_without_a_readable_table(tmp_path, workbook, message): + yoy, gap_dir = write_tree(tmp_path) + before = {path: path.read_text() for path in [yoy, *gap_dir.iterdir()]} + with pytest.raises(ValueError, match=message): + run_main(tmp_path, workbook, "--yaml-path", str(yoy)) + assert {path: path.read_text() for path in before} == before + + +def test_main_can_leave_the_national_living_wage_alone(tmp_path): + yoy, _ = write_tree(tmp_path) + no_table = rebuild_workbook(make_test_xlsx(), drop={"1.13"}) + assert ( + run_main( + tmp_path, + no_table, + "--yaml-path", + str(yoy), + "--skip-national-living-wage", + ) + == 0 + ) + content = yoy.read_text() + assert "2026-01-01: 0.0248" in content + assert LEVEL_SERIES_FIXTURE in content + + +NLW_CHANGES = st.floats(min_value=-0.05, max_value=0.10, allow_nan=False) +# average_earnings from 2010, so a table ending in any year has a long-run +# path after it. +EARLY_EARNINGS_FIXTURE = YOY_GROWTH_FIXTURE.replace( + " average_earnings:\n values:\n 2024-01-01:", + " average_earnings:\n values:\n 2010-01-01:", +) + + +@settings(max_examples=60, deadline=None, derandomize=True) +@given( + first_level=st.floats(min_value=5, max_value=20, allow_nan=False), + changes=st.lists(NLW_CHANGES, min_size=1, max_size=11), + first_year=st.integers(2015, 2027), + data=st.data(), +) +def test_the_model_index_reproduces_the_published_levels( + first_level, changes, first_year, data +): + """For any national living wage-like table (£5 to £20 in its first year, + up to 12 years, each changing by -5% to +10%): each year's growth is its + level over the year before's, to six decimals; the years up to the one + the note names fall under the rates set and the rest under the OBR's + estimates; and the index the model compounds from the written series is + above zero and carries the table's second-year level to within half a + penny of every later level.""" + levels = [first_level] + for change in changes: + levels.append(levels[-1] * (1 + change)) + years = range(first_year, first_year + len(levels)) + published = dict(zip(years, levels)) + set_until = data.draw(st.integers(first_year, years[-1])) + workbook = nlw_workbook(nlw_rows(tuple(levels), first_year, set_until)) + growth = compute_level_growth(workbook)[NLW_PATH] + assert growth.line.levels == published + for year in years[1:]: + assert growth.values[year] == round( + published[year] / published[year - 1] - 1, 6 + ) + + rendered = render_nlw(EARLY_EARNINGS_FIXTURE, workbook) + values = _values(rendered, NLW_PATH) + index = model_index(values) + assert all(level > 0 for level in index.values()) + base = years[1] + for year in years[1:]: + carried = published[base] * index[year] / index[base] + assert abs(carried - published[year]) < 0.005, year + + block = rendered[rendered.index(" national_living_wage:") :] + kind = None + for line in block.split("\n")[: block.split("\n").index(" metadata:")]: + if line.strip().startswith("# Set by the Government"): + kind = "set" + elif line.strip().startswith("# OBR estimates"): + kind = "estimate" + elif line.strip().startswith("# Beyond"): + kind = None + elif (match := import_obr_forecasts.VALUE_LINE_RE.match(line)) and int( + match.group(1) + ) in growth.values: + year = int(match.group(1)) + assert kind == ("set" if year <= set_until else "estimate"), year + + +TINY_TO_HUGE = st.one_of( + st.floats(min_value=1e-9, max_value=1e9, allow_nan=False), + st.floats(min_value=1e-300, max_value=1e-200, allow_nan=False), + st.floats(min_value=1e200, max_value=1e300, allow_nan=False), +) + + +@settings(max_examples=60, deadline=None, derandomize=True) +@given(levels=st.lists(TINY_TO_HUGE, min_size=2, max_size=8)) +def test_a_written_level_series_keeps_every_index_positive(levels): + """For any positive levels, however far apart: the importer refuses + exactly when the six-decimal growth takes the five-decimal index the + model compounds through the table (from 1 in its first year) to zero or + past every float. Otherwise the index of the series it writes, with + average_earnings after the table, stays above zero in every year.""" + workbook = nlw_workbook(nlw_rows(tuple(levels), 2023, 2023)) + values = { + 2023 + i: round(levels[i] / levels[i - 1] - 1, 6) + 0.0 + for i in range(1, len(levels)) + } + through_table = model_index({2023: 0.0, **values}) + compounds = all( + math.isfinite(through_table[year]) and through_table[year] > 0 + for year in values + ) + try: + growth = compute_level_growth(workbook)[NLW_PATH] + except ValueError as error: + event("refused") + assert "which the model cannot compound from" in str(error) + assert not compounds + return + event("written") + assert compounds + assert growth.values == values + written = model_index( + _values(render_nlw(EARLY_EARNINGS_FIXTURE, workbook), NLW_PATH) + ) + assert all(math.isfinite(level) and level > 0 for level in written.values()) diff --git a/policyengine_uk/tests/test_minimum_wage_projection.py b/policyengine_uk/tests/test_minimum_wage_projection.py index 19bad37271..52be1d17c5 100644 --- a/policyengine_uk/tests/test_minimum_wage_projection.py +++ b/policyengine_uk/tests/test_minimum_wage_projection.py @@ -45,6 +45,7 @@ process_parameter_tree, statutory_parameters, ) +from policyengine_uk.utils.import_obr_forecasts import compute_level_growth PARAMETER_DIR = ( Path(__file__).resolve().parent.parent / "parameters/gov/hmrc/minimum_wage" @@ -106,7 +107,8 @@ def test_rates_are_the_last_rate_set_uprated_by_the_index(year): # OBR EFO October 2026, detailed forecast tables (economy), Table 1.13: the # national living wage in April of each year, £ an hour. "The NLW has been set # by the Government up until 2027 ... Figures beyond 2027 are OBR estimates -# consistent with the rest of our economy forecast." +# consistent with the rest of our economy forecast." The table is also in +# tests/fixtures/obr_economy, which the importer reads. OBR_NATIONAL_LIVING_WAGE = { 2020: 8.72, 2021: 8.91, @@ -148,6 +150,39 @@ def test_the_series_reproduces_the_obr_table(): assert wage == pytest.approx(round(rate, 2), abs=1e-9), year +TABLE_1_13 = ( + Path(__file__).parent + / "fixtures/obr_economy/efo_october_2026_economy_table_1_13.xlsx" +) + + +def test_the_table_here_is_the_one_the_importer_reads(): + """OBR_NATIONAL_LIVING_WAGE is Table 1.13 as the importer reads it from + the real October 2026 workbook, so a refreshed fixture fails here until + the table here moves with it.""" + line = compute_level_growth(TABLE_1_13.read_bytes())[("national_living_wage",)].line + assert line.levels == OBR_NATIONAL_LIVING_WAGE + assert line.set_until == 2027 + + +def test_the_index_carries_any_year_of_the_table_to_any_later_year(): + """Invariants of the series: the national living wage index is above + zero in every year to 2040, and from any year of Table 1.13 from its + second (the index's first) the index's growth carries that year's level + to every later year's in the table, to the penny.""" + p = system.parameters + first = min(d.year for d in GROWTH["national_living_wage"]["values"]) + assert first == min(OBR_NATIONAL_LIVING_WAGE) + 1 + for year in range(first, LAST_CHECKED + 1): + assert _index(p, year) > 0, year + for base in range(first, max(OBR_NATIONAL_LIVING_WAGE) + 1): + for year in range(base, max(OBR_NATIONAL_LIVING_WAGE) + 1): + carried = OBR_NATIONAL_LIVING_WAGE[base] * _index(p, year) / _index(p, base) + assert round(carried, 2) == pytest.approx( + round(OBR_NATIONAL_LIVING_WAGE[year], 2), abs=1e-9 + ), (base, year) + + def test_after_the_efo_the_series_is_average_earnings_growth(): """The Low Pay Commission's reference point is two-thirds of median hourly earnings (remit 2026), so after the OBR's last estimate the series takes diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 7616e8d63f..b60f3b9713 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1664,20 +1664,6 @@ def find_level_line( ) header = year_rows[0] years = _year_columns(rows[header]) - units = [ - normalise_label(value) - for row in rows[:header] - for value in row.values() - if normalise_label(value) - ] - if units != [spec.unit]: - raise ValueError( - f"{where} must declare its unit as {spec.unit!r} above its years; " - f"found {units}" - ) - ordered = [years[column] for column in sorted(years, key=column_index)] - if ordered != list(range(ordered[0], ordered[0] + len(ordered))): - raise ValueError(f"{where}'s years skip, repeat or run backwards: {ordered}") first_year_column = min(column_index(column) for column in years) matches = [] for index, row in enumerate(rows): @@ -1697,6 +1683,20 @@ def find_level_line( index, label, row = matches[0] if index < header: raise ValueError(f'{where} has its "{spec.row}" row above its years') + units = [ + normalise_label(value) + for header_row in rows[:header] + for value in header_row.values() + if normalise_label(value) + ] + if units != [spec.unit]: + raise ValueError( + f"{where} must declare its unit as {spec.unit!r} above its years; " + f"found {units}" + ) + ordered = [years[column] for column in sorted(years, key=column_index)] + if ordered != list(range(ordered[0], ordered[0] + len(ordered))): + raise ValueError(f"{where}'s years skip, repeat or run backwards: {ordered}") stray = sorted( column for column, value in row.items() From a40bb69363480ee601548dfef8fdd80f9dd623c3 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 05:50:03 -0400 Subject: [PATCH 03/22] Document the national living wage import The economic assumptions README describes how the importer reads Table 1.13 and what it refuses, adds the refresh step, and drops the notes that the series is kept by hand. The fixture's README names its source and hash. The changelog fragment and docstrings name who acts. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2721.added.md | 1 + .../gov/economic_assumptions/README.md | 49 ++++++++++++------- .../tests/fixtures/obr_economy/README.md | 22 +++++++++ .../tests/test_import_obr_forecasts.py | 8 +-- .../tests/test_minimum_wage_projection.py | 2 +- policyengine_uk/utils/import_obr_forecasts.py | 27 +++++----- 6 files changed, 74 insertions(+), 35 deletions(-) create mode 100644 changelog.d/2721.added.md create mode 100644 policyengine_uk/tests/fixtures/obr_economy/README.md diff --git a/changelog.d/2721.added.md b/changelog.d/2721.added.md new file mode 100644 index 0000000000..6f0578ca39 --- /dev/null +++ b/changelog.d/2721.added.md @@ -0,0 +1 @@ +Taught the OBR forecasts importer to write the national living wage series from the EFO economy tables' national living wage table (Table 1.13), reading its levels across the row of years and refusing any other layout, so the minimum wage projection moves with each EFO and no one keeps the series by hand. diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index ab0173d409..0143767519 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -62,15 +62,24 @@ assumes, and the approximations. `national_living_wage` projects the minimum wage rates (`gov.hmrc.minimum_wage`) after the last rate set. It is the growth of the -rates set to 2027, then the OBR's estimates in Table 1.13 of the detailed -economy tables, then `average_earnings`' long-run growth. The checked-in -estimates are the October 2026 EFO's, ahead of the other series' March 2026 -vintage, because HM Treasury costed Budget 2026's Carer's Allowance taper on -them. `utils/import_obr_forecasts.py` doesn't read Table 1.13, so the series -is updated by hand with each EFO. `test_minimum_wage_projection.py` holds the -table and fails if the series stops reproducing it, or if its long-run years -stop matching `average_earnings`. Each rate is set by an S.I., so this is a projection, not -a formula. +rates the Government has set, then of the OBR's estimates, both from the +national living wage table of the detailed economy tables (Table 1.13 in +October 2026), then `average_earnings`' long-run growth, year for year. HM +Treasury costed Budget 2026's Carer's Allowance taper on the OBR's estimates. +`utils/import_obr_forecasts.py` writes the series with the other economy +series. It finds the table by its title, reads the level under each year +across the table's one row of years, and writes each year's level over the +year before's, to six decimals. The table's note says up to which year the +Government has set the rate, and the series' comments keep those years apart +from the OBR's estimates. The importer stops before writing any file if the +table declares a unit other than £ per hour, has more or fewer than one row of +years or one row labelled "National Living Wage (NLW)", has years that skip or +repeat, lacks a positive level under a year or has a figure with no year above +it, or lacks that note. `test_minimum_wage_projection.py` holds the October +2026 table, checks it against the importer's fixture, and fails if the series +stops reproducing the table to the penny, or if its long-run years stop +matching `average_earnings`. An S.I. sets each rate, so this is a +projection, not a formula. ## Time horizons @@ -483,9 +492,17 @@ the forecast levels, though not their growth. and extend the Machine Games Duty measure's receipts when the forecast runs past 2031-32. Then update the published levels in `policyengine_uk/tests/test_gambling_duty_base_projection.py`. -7. If the new EFO runs past `PROJECTION_END_YEAR`, move it on (see +7. The same run refreshes `national_living_wage` from the new economy + tables' national living wage table (Table 1.13 in October 2026), with + `average_earnings` after it. Then replace the cut-down table in + `policyengine_uk/tests/fixtures/obr_economy/` (see its README) and the + levels in `OBR_NATIONAL_LIVING_WAGE` in + `policyengine_uk/tests/test_minimum_wage_projection.py`. Pass + `--skip-national-living-wage` only to leave the series on the previous + EFO. +8. If the new EFO runs past `PROJECTION_END_YEAR`, move it on (see [Dataset projection horizon](#dataset-projection-horizon)). -8. Refresh the household energy price inputs (see +9. Refresh the household energy price inputs (see [Household energy prices](#household-energy-prices)) from the new EFO's economy tables, the MM23 release of the same month and the latest Energy Trends tables. The importer records the last quarter ONS publishes in @@ -538,12 +555,10 @@ Bank Rate (`gov.boe.base_rate`, economy Table 1.9) and households' financial assets (`household.wealth.financial_assets`, Table 1.11) also take October 2026 values. -The national living wage forecast (economy Table 1.13) is not read here. Until -PolicyEngine/policyengine-uk#2700 adds it as `obr.national_living_wage`, the -minimum wage rates, and the rules built on them, stay at their April 2026 -levels; #2721 tracks teaching the importer to read the table. Whichever of -#2700 and the October 2026 update merges second copies the 2032+ earnings path -into that series. +The importer did not read the national living wage table (economy Table +1.13) when PolicyEngine/policyengine-uk#2700 added `obr.national_living_wage`, +so maintainers first copied the series from the October 2026 table by hand. +Rerunning the importer on that table reproduces every hand-kept value (#2721). The EFO refresh does not touch `finance_ni.domestic_rates`; OBR does not publish a Northern Ireland domestic rates bill series (see below). diff --git a/policyengine_uk/tests/fixtures/obr_economy/README.md b/policyengine_uk/tests/fixtures/obr_economy/README.md new file mode 100644 index 0000000000..fb592cc01a --- /dev/null +++ b/policyengine_uk/tests/fixtures/obr_economy/README.md @@ -0,0 +1,22 @@ +# Real OBR economy table fixture + +`efo_october_2026_economy_table_1_13.xlsx` holds the Contents and 1.13 +(National Living Wage) worksheets of the OBR October 2026 EFO detailed +forecast tables, economy +(`efo-october-2026-detailed-forecast-tables-economy.xlsx` in the detailed +forecast tables zip file, SHA-256 +`96795d1747bb5a4978c7fb768689d5fd31f568e5223db8a94b1bd92ba7f6c118`). Source: +. + +As for the energy price fixtures, the cut kept only those worksheets in the +workbook's sheet list and relationships and dropped its defined names. The +worksheet XML and shared strings are the originals, byte for byte. The file is +a minimal archive for the importer's XLSX reader, not a complete Excel file. + +`policyengine_uk/utils/import_obr_forecasts.py` must reproduce +`obr.national_living_wage` in `yoy_growth.yaml` from it byte for byte. Tests +change cells of sheet 1.13 to check that the importer refuses a changed layout +rather than reading another row. No test downloads a workbook. With each new +EFO, replace the file with the new table and update `OBR_NATIONAL_LIVING_WAGE` +in `policyengine_uk/tests/test_minimum_wage_projection.py`, which must equal +it. diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 6145da288c..df198ebdae 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -2697,8 +2697,8 @@ def test_october_2026_table_1_13_reproduces_the_committed_series(): def edit_real_table_1_13(edits): """The real October 2026 workbook with cells of sheet 1.13 set: text as - an inline string, a number as a number, None to empty the cell. Every - other archive entry is unchanged.""" + an inline string, a number as a number, None to empty the cell. It + copies every other archive entry as it was.""" output = BytesIO() with ZipFile(BytesIO(REAL_TABLE_1_13)) as source, ZipFile(output, "w") as dest: path = _sheet_paths(source)["1.13"] @@ -2973,8 +2973,8 @@ def test_the_model_index_reproduces_the_published_levels( up to 12 years, each changing by -5% to +10%): each year's growth is its level over the year before's, to six decimals; the years up to the one the note names fall under the rates set and the rest under the OBR's - estimates; and the index the model compounds from the written series is - above zero and carries the table's second-year level to within half a + estimates; and the index the model compounds from the written series + exceeds zero and carries the table's second-year level to within half a penny of every later level.""" levels = [first_level] for change in changes: diff --git a/policyengine_uk/tests/test_minimum_wage_projection.py b/policyengine_uk/tests/test_minimum_wage_projection.py index 52be1d17c5..daa251162e 100644 --- a/policyengine_uk/tests/test_minimum_wage_projection.py +++ b/policyengine_uk/tests/test_minimum_wage_projection.py @@ -166,7 +166,7 @@ def test_the_table_here_is_the_one_the_importer_reads(): def test_the_index_carries_any_year_of_the_table_to_any_later_year(): - """Invariants of the series: the national living wage index is above + """Invariants of the series: the national living wage index exceeds zero in every year to 2040, and from any year of Table 1.13 from its second (the index's first) the index's growth carries that year's level to every later year's in the table, to the penny.""" diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index b60f3b9713..79af6857df 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1551,14 +1551,15 @@ class LevelSeriesSpec: """An economy table that publishes a series as levels across a row, not as growth down a column. - ``path`` is the series' place under ``obr``. The table is found by its - title, as receipts tables are. Its header declares one unit, ``unit``; - one row of cells holds consecutive calendar years; and one row below it, - labelled ``row``, holds a level under each year and nowhere else. - Titles, units and labels are compared lowercased, without footnote - markers. A note that the rate "has been set by the Government up until" - a year, and that figures beyond that year "are OBR estimates", splits - the years the Government has set from the OBR's estimates. + ``path`` is the series' place under ``obr``. The importer finds the + table by its title, as it finds receipts tables. The table's header + declares one unit, ``unit``; one row of cells holds consecutive calendar + years; and one row below it, labelled ``row``, holds a level under each + year and nowhere else. The importer compares titles, units and labels + lowercased, without footnote markers. A note that the rate "has been set + by the Government up until" a year, and that figures beyond that year + "are OBR estimates", splits the years the Government has set from the + OBR's estimates. """ path: tuple[str, ...] @@ -1641,7 +1642,7 @@ def find_level_line( """The levels in the one row labelled ``spec.row`` of the one table titled ``spec.title``, read across its row of years. - Any other layout is refused rather than read: another unit, a second + It refuses any other layout rather than read it: another unit, a second row of years or of the label, a level missing under a year or one with no year above it, or years that skip or repeat. """ @@ -1837,10 +1838,10 @@ def render_level_series_yaml( """Rewrite each level series' values and first reference in a yoy_growth.yaml. - Years before the table's second year are kept, so the series' index - keeps its base year, and its other references stay. Years after the - table follow average_earnings as ``content`` holds it. The result is - checked to differ from ``content`` only in those values and that + It keeps the years before the table's second year, so the series' index + keeps its base year, and the series' other references. Years after the + table follow average_earnings as ``content`` holds it. It checks that + the result differs from ``content`` only in those values and that reference. """ month, year = release From 2ed288c411017b81a064a70bf0842bf71040248e Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 06:10:20 -0400 Subject: [PATCH 04/22] Check the finished national living wage index before writing it render_level_series_yaml now runs #2738's check_index_stays_positive on the finished series, with its kept years and the average_earnings tail, as the receipts series do. The tests add a tail that takes the index to zero, growth past every float, and a property whose falling tail exercises the finished check. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 100 +++++++++++++----- policyengine_uk/utils/import_obr_forecasts.py | 5 +- 2 files changed, 75 insertions(+), 30 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index a0d3c8e254..43c18bde66 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -9,7 +9,7 @@ import pytest import yaml -from hypothesis import event, given, settings +from hypothesis import event, example, given, settings from hypothesis import strategies as st from policyengine_core.parameters import ParameterNode @@ -2986,12 +2986,17 @@ def test_main_can_leave_the_national_living_wage_alone(tmp_path): NLW_CHANGES = st.floats(min_value=-0.05, max_value=0.10, allow_nan=False) -# average_earnings from 2010, so a table ending in any year has a long-run -# path after it. -EARLY_EARNINGS_FIXTURE = YOY_GROWTH_FIXTURE.replace( - " average_earnings:\n values:\n 2024-01-01:", - " average_earnings:\n values:\n 2010-01-01:", -) + + +def earnings_fixture(rate): + """The fixture with average_earnings from 2010, so a table ending in any + year has a long-run path after it, at ``rate`` where it holds 2%.""" + flat = FLAT_SERIES_FIXTURE.format(key="average_earnings") + early = flat.replace("2024-01-01", "2010-01-01").replace("0.0200", f"{rate:.4f}") + return YOY_GROWTH_FIXTURE.replace(flat, early) + + +EARLY_EARNINGS_FIXTURE = earnings_fixture(0.02) @settings(max_examples=60, deadline=None, derandomize=True) @@ -3050,42 +3055,81 @@ def test_the_model_index_reproduces_the_published_levels( assert kind == ("set" if year <= set_until else "estimate"), year +def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): + """The table keeps the index above zero; the average_earnings tail takes + it there, so the finished series' check stops the render.""" + workbook = nlw_workbook(nlw_rows((10.0, 10.0, 0.0001), 2023, 2023)) + render_nlw(workbook=workbook) + flat = FLAT_SERIES_FIXTURE.format(key="average_earnings") + content = YOY_GROWTH_FIXTURE.replace(flat, flat.replace("0.0200", "-0.5100")) + assert content != YOY_GROWTH_FIXTURE + with pytest.raises( + ValueError, match="national_living_wage: the index reaches 0.0 in 2031" + ): + render_nlw(content, workbook) + + +def test_growth_past_every_float_is_refused(): + with pytest.raises(ValueError, match="takes the index to inf"): + compute_level_growth(nlw_workbook(nlw_rows((1e-300, 1e300), 2023, 2023))) + + +# Levels whose ratios stay within a float: the index tracks each level over +# the first, at most 1e300, so only a fall can stop it. TINY_TO_HUGE = st.one_of( + st.floats(min_value=0.01, max_value=1, allow_nan=False), st.floats(min_value=1e-9, max_value=1e9, allow_nan=False), - st.floats(min_value=1e-300, max_value=1e-200, allow_nan=False), - st.floats(min_value=1e200, max_value=1e300, allow_nan=False), + st.floats(min_value=1e-100, max_value=1e-50, allow_nan=False), + st.floats(min_value=1e50, max_value=1e200, allow_nan=False), ) @settings(max_examples=60, deadline=None, derandomize=True) -@given(levels=st.lists(TINY_TO_HUGE, min_size=2, max_size=8)) -def test_a_written_level_series_keeps_every_index_positive(levels): - """For any positive levels, however far apart: the importer refuses - exactly when the six-decimal growth takes the five-decimal index the - model compounds through the table (from 1 in its first year) to zero or - past every float. Otherwise the index of the series it writes, with - average_earnings after the table, stays above zero in every year.""" +@given( + levels=st.lists(TINY_TO_HUGE, min_size=2, max_size=8), + earnings=st.sampled_from((0.02, -0.51)), +) +# A fall to a thousandth passes the table's check; the falling tail then +# takes the finished index to zero in 2037. +@example(levels=[1.0, 0.001], earnings=-0.51) +def test_a_written_level_series_keeps_every_index_positive(levels, earnings): + """For any positive levels, however far apart: the importer refuses the + table exactly when its six-decimal growth takes the five-decimal index + the model compounds through it (from 1 in its first year) to zero, and + refuses to write the series exactly when the index of the finished + series, with the year kept before it and average_earnings (rising 2% or + falling 51% a year) after it, reaches zero. Otherwise that index stays + above zero in every year.""" + content = earnings_fixture(earnings) workbook = nlw_workbook(nlw_rows(tuple(levels), 2023, 2023)) values = { 2023 + i: round(levels[i] / levels[i - 1] - 1, 6) + 0.0 for i in range(1, len(levels)) } through_table = model_index({2023: 0.0, **values}) - compounds = all( - math.isfinite(through_table[year]) and through_table[year] > 0 - for year in values - ) try: growth = compute_level_growth(workbook)[NLW_PATH] except ValueError as error: - event("refused") + event("refused: table") assert "which the model cannot compound from" in str(error) - assert not compounds + assert any(through_table[year] <= 0 for year in values) return - event("written") - assert compounds + assert all(through_table[year] > 0 for year in values) assert growth.values == values - written = model_index( - _values(render_nlw(EARLY_EARNINGS_FIXTURE, workbook), NLW_PATH) - ) - assert all(math.isfinite(level) and level > 0 for level in written.values()) + with pytest.MonkeyPatch.context() as patch: + patch.setattr( + import_obr_forecasts, "check_index_stays_positive", lambda *_: None + ) + unchecked = render_nlw(content, workbook) + finished = model_index(_values(unchecked, NLW_PATH)) + assert all(math.isfinite(level) for level in finished.values()) + try: + rendered = render_nlw(content, workbook) + except ValueError as error: + event("refused: finished series") + assert "national_living_wage: the index reaches" in str(error) + assert any(level <= 0 for level in finished.values()) + return + event("written") + assert rendered == unchecked + assert all(level > 0 for level in finished.values()) diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index e6c35a53fb..62a6744b50 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1860,8 +1860,8 @@ def render_level_series_yaml( It keeps the years before the table's second year, so the series' index keeps its base year, and the series' other references. Years after the table follow average_earnings as ``content`` holds it. It checks that - the result differs from ``content`` only in those values and that - reference. + the finished series' index stays above zero, and that the result + differs from ``content`` only in those values and that reference. """ month, year = release expected = copy.deepcopy(yaml.safe_load(content)) @@ -1919,6 +1919,7 @@ def render_level_series_yaml( for entry_year, value in {**series.values, **tail}.items() }, } + check_index_stays_positive(path, node["values"]) node["metadata"]["reference"][0] = reference rendered = "\n".join(lines) From ea84590c544e8d57c045b137f0b25285f2fea1a5 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 07:12:51 -0400 Subject: [PATCH 05/22] Check the level series' index to the end of the growth file The positivity property now compounds the finished series as the model's rule states, through LAST_INDEX_YEAR as #2738's check does, and checks that create_economic_assumption_indices agrees with it as far as the model compounds. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 22 ++++++++++++++++--- 1 file changed, 19 insertions(+), 3 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 21e754effc..3240652191 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -2662,6 +2662,17 @@ def model_index(values): return {year: index(f"{year}-01-01") for year in range(min(values), 2040)} +def compounded(values, last_year): + """A growth series' index as the model's rule states it: 1 in the first + listed year, then each January's value in force, to five decimals.""" + years = sorted(values) + index = {years[0]: 1.0} + for year in range(years[0] + 1, last_year + 1): + in_force = values[max(y for y in years if y <= year)] + index[year] = round(index[year - 1] * (1 + in_force), 5) + return index + + def test_level_growth_is_the_ratio_of_consecutive_levels(): growth = compute_level_growth(nlw_workbook())[NLW_PATH] assert growth.line.levels == dict(zip(range(2023, 2028), NLW_LEVELS)) @@ -3111,8 +3122,9 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): the model compounds through it (from 1 in its first year) to zero, and refuses to write the series exactly when the index of the finished series, with the year kept before it and average_earnings (rising 2% or - falling 51% a year) after it, reaches zero. Otherwise that index stays - above zero in every year.""" + falling 51% a year) after it, reaches zero by the end of the growth + file. Otherwise that index stays above zero in every year, and the + model's index agrees with it as far as the model compounds.""" content = earnings_fixture(earnings) workbook = nlw_workbook(nlw_rows(tuple(levels), 2023, 2023)) values = { @@ -3134,7 +3146,11 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): import_obr_forecasts, "check_index_stays_positive", lambda *_: None ) unchecked = render_nlw(content, workbook) - finished = model_index(_values(unchecked, NLW_PATH)) + written = _values(unchecked, NLW_PATH) + finished = compounded(written, import_obr_forecasts.LAST_INDEX_YEAR) + # The model compounds the same index, as far as 2039. + model = model_index(written) + assert {year: finished[year] for year in model} == model assert all(math.isfinite(level) for level in finished.values()) try: rendered = render_nlw(content, workbook) From b25ac1a7147f220388219ced199c7eff3dd4077f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 09:10:50 -0400 Subject: [PATCH 06/22] Name who acts in the national living wage notes The YAML comment makes the importer and the government the actors and quotes the 2026 remit to the Low Pay Commission. The README paragraph drops its be-verb spines and the history note about the series once being copied by hand. The fixture README and changelog fragment name who acts. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2721.added.md | 2 +- .../gov/economic_assumptions/README.md | 26 +++++++------------ .../gov/economic_assumptions/yoy_growth.yaml | 18 +++++++------ .../tests/fixtures/obr_economy/README.md | 9 ++++--- 4 files changed, 26 insertions(+), 29 deletions(-) diff --git a/changelog.d/2721.added.md b/changelog.d/2721.added.md index 6f0578ca39..3644cac72e 100644 --- a/changelog.d/2721.added.md +++ b/changelog.d/2721.added.md @@ -1 +1 @@ -Taught the OBR forecasts importer to write the national living wage series from the EFO economy tables' national living wage table (Table 1.13), reading its levels across the row of years and refusing any other layout, so the minimum wage projection moves with each EFO and no one keeps the series by hand. +Taught the OBR forecasts importer to write the national living wage series from the EFO economy tables' national living wage table (Table 1.13). It reads the levels across the table's row of years, refuses any other layout, and refreshes the minimum wage projection with each EFO. diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index 4ac9f668ba..819321a252 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -61,7 +61,7 @@ for each duty's rates, the higher-stake share Machine Games Duty's rate assumes, and the approximations. `national_living_wage` projects the minimum wage rates -(`gov.hmrc.minimum_wage`) after the last rate set. It is the growth of the +(`gov.hmrc.minimum_wage`) after the last rate set. It holds the growth of the rates the Government has set, then of the OBR's estimates, both from the national living wage table of the detailed economy tables (Table 1.13 in October 2026), then `average_earnings`' long-run growth, year for year. HM @@ -70,16 +70,15 @@ Treasury costed Budget 2026's Carer's Allowance taper on the OBR's estimates. series. It finds the table by its title, reads the level under each year across the table's one row of years, and writes each year's level over the year before's, to six decimals. The table's note says up to which year the -Government has set the rate, and the series' comments keep those years apart -from the OBR's estimates. The importer stops before writing any file if the -table declares a unit other than £ per hour, has more or fewer than one row of -years or one row labelled "National Living Wage (NLW)", has years that skip or -repeat, lacks a positive level under a year or has a figure with no year above -it, or lacks that note. `test_minimum_wage_projection.py` holds the October -2026 table, checks it against the importer's fixture, and fails if the series -stops reproducing the table to the penny, or if its long-run years stop -matching `average_earnings`. An S.I. sets each rate, so this is a -projection, not a formula. +Government has set the rate, and the importer writes those years and the +OBR's estimates under separate comments. The importer stops before writing +any file if the table declares a unit other than £ per hour, or lacks one row +of years and one row labelled "National Living Wage (NLW)". It also stops if +the years skip or repeat, if a year lacks a positive level, if a figure has no +year above it, or if the table lacks that note. `test_minimum_wage_projection.py` +holds the October 2026 table, checks it against the importer's fixture, and +fails if the series stops reproducing the table to the penny, or if its +long-run years stop matching `average_earnings`. ## Time horizons @@ -554,11 +553,6 @@ Bank Rate (`gov.boe.base_rate`, economy Table 1.9) and households' financial assets (`household.wealth.financial_assets`, Table 1.11) also take October 2026 values. -The importer did not read the national living wage table (economy Table -1.13) when PolicyEngine/policyengine-uk#2700 added `obr.national_living_wage`, -so maintainers first copied the series from the October 2026 table by hand. -Rerunning the importer on that table reproduces every hand-kept value (#2721). - The EFO refresh does not touch `finance_ni.domestic_rates`; OBR does not publish a Northern Ireland domestic rates bill series (see below). diff --git a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml index 2726ad906b..480820b8d6 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml +++ b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml @@ -212,14 +212,16 @@ obr: # Government has set, then "OBR estimates consistent with the rest of our # economy forecast". utils/import_obr_forecasts.py reads the table and # writes each year's rate over the year before's, to six decimals, so - # compounding them reproduces the table to the penny. Its comments below - # keep the years the Government has set apart from the OBR's estimates; - # gov.hmrc.minimum_wage cites the regulations or Budget behind each rate - # set. HM Treasury costed the Budget 2026 Carer's Allowance taper on these - # "OBR projections for increases in the National Living Wage" (policy - # costings, October 2026). After the EFO horizon the series takes - # average_earnings' long-run growth: the Low Pay Commission's reference - # point for the rate is two-thirds of median hourly earnings (remit 2026). + # compounding them reproduces the table to the penny. It writes the years + # the Government has set and the OBR's estimates under separate comments + # below; gov.hmrc.minimum_wage cites the regulations or Budget behind + # each rate set. HM Treasury costed the Budget 2026 Carer's Allowance + # taper on these "OBR projections for increases in the National Living + # Wage" (policy costings, October 2026). After the EFO horizon the + # importer copies average_earnings' long-run growth: in its 2026 remit to + # the Low Pay Commission, the government says two-thirds of median hourly + # earnings "should continue to be the key reference point for future + # rates". values: # Set by the Government (2021-2027) from OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)": £8.72 in 2020 to £13.21 in 2027 2021-01-01: 0.021789 diff --git a/policyengine_uk/tests/fixtures/obr_economy/README.md b/policyengine_uk/tests/fixtures/obr_economy/README.md index fb592cc01a..29f9210591 100644 --- a/policyengine_uk/tests/fixtures/obr_economy/README.md +++ b/policyengine_uk/tests/fixtures/obr_economy/README.md @@ -8,10 +8,11 @@ forecast tables zip file, SHA-256 `96795d1747bb5a4978c7fb768689d5fd31f568e5223db8a94b1bd92ba7f6c118`). Source: . -As for the energy price fixtures, the cut kept only those worksheets in the -workbook's sheet list and relationships and dropped its defined names. The -worksheet XML and shared strings are the originals, byte for byte. The file is -a minimal archive for the importer's XLSX reader, not a complete Excel file. +As with the energy price fixtures, a script kept only those worksheets in the +workbook's sheet list and relationships, dropped its defined names, and copied +the worksheet XML and shared strings byte for byte. The archive serves the +importer's XLSX reader and leaves out the styles and other parts a full Excel +file has. `policyengine_uk/utils/import_obr_forecasts.py` must reproduce `obr.national_living_wage` in `yoy_growth.yaml` from it byte for byte. Tests From 5bae36a48d247284b329e0fa4d8b7b7e40f05341 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 11:20:33 -0400 Subject: [PATCH 07/22] Keep each national living wage estimate's penny through the model's path The importer now compounds the finished series as create_economic_assumption_indices does and projects each OBR estimate from the table's level in the last year the Government has set, as uprate_parameters does for gov.hmrc.minimum_wage, rounding to the nearest penny. Where a year's six-decimal ratio of levels projects another penny, it writes the six-decimal growth from the level the index carries the rate to in the year before, or the nearest six-decimal growth to that which keeps the penny. It refuses a year no six-decimal growth can reproduce, a table whose index starts after the year set, and one that runs past the model's indices. The committed series comes out byte for byte as before. It also refuses a table whose statements of the year set, or of the year beyond which figures are OBR estimates, name different years; reads past a footnote marker after a closing bracket or in superscript; and the finished-index check refuses an index that overflows as well as one that reaches zero. Tests: the half-penny property becomes equality after penny rounding through the model's own code; the review's two counterexamples, the refusals, a differential of the importer's projection against the model's, the rounding the minimum wage parameters use, the conflicting notes, the footnote markers, the overflowing index and a note naming a year before the table. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 344 ++++++++++++++++-- .../tests/test_minimum_wage_projection.py | 4 +- policyengine_uk/utils/import_obr_forecasts.py | 281 +++++++++++--- 3 files changed, 540 insertions(+), 89 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 3240652191..eb757acc25 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -12,6 +12,9 @@ from hypothesis import event, example, given, settings from hypothesis import strategies as st from policyengine_core.parameters import ParameterNode +from policyengine_core.parameters.operations.uprate_parameters import ( + uprate_parameters, +) from policyengine_uk.parameters.gov.economic_assumptions.create_economic_assumption_indices import ( create_economic_assumption_indices, @@ -31,12 +34,14 @@ average_earnings_years, compute_level_growth, compute_receipts_growth, + compound_model_index, find_receipts_line, format_gap, format_growth, get_yoy_growth_path, infer_workbook_release, main, + projected_rate_units, read_calendar_values, read_workbook_sheets, render_forecast_gap_yaml, @@ -2662,6 +2667,48 @@ def model_index(values): return {year: index(f"{year}-01-01") for year in range(min(values), 2040)} +MINIMUM_WAGE_DIR = Path(__file__).parents[1] / "parameters/gov/hmrc/minimum_wage" +# The national living wage's uprating metadata: the index, and the rounding to +# the nearest penny. +NLW_UPRATING = yaml.safe_load((MINIMUM_WAGE_DIR / "non_apprentice.yaml").read_text())[ + "brackets" +][-1]["amount"]["metadata"]["uprating"] + + +def model_wages(values, set_until, rate): + """The national living wage the model projects in April of each year + after ``set_until`` to 2039, from ``rate`` set in April ``set_until``: the + growth series through create_economic_assumption_indices, then a rate + with the national living wage's uprating metadata through + policyengine-core's uprate_parameters.""" + tree = ParameterNode( + "", + data={ + "gov": { + "economic_assumptions": { + "yoy_growth": { + "obr": { + "national_living_wage": { + "values": { + f"{year}-01-01": value + for year, value in values.items() + } + } + } + } + } + }, + "wage": { + "metadata": {"uprating": NLW_UPRATING}, + "values": {f"{set_until}-04-01": rate}, + }, + }, + ) + create_economic_assumption_indices(tree) + uprate_parameters(tree) + return {year: tree.wage(f"{year}-04-01") for year in range(set_until + 1, 2040)} + + def compounded(values, last_year): """A growth series' index as the model's rule states it: 1 in the first listed year, then each January's value in force, to five decimals.""" @@ -2722,9 +2769,10 @@ def test_the_table_must_appear_once(): 2028: "13.513623297578135", 2029: "13.854853379339588", 2030: "14.23293930734993", 2031: "14.617023376183901", } # fmt: skip -# The series as maintainers kept it by hand before the importer read the -# table (#2700): every year is its level over the year before's. -HAND_KEPT_NLW_GROWTH = { +# The growth the importer must write from OCTOBER_2026_NLW: each year's level +# over the year before's, less one, to six decimals, written out by hand. No +# year's ratio projects another penny, so every year keeps it. +OCTOBER_2026_NLW_GROWTH = { 2021: 0.021789, 2022: 0.066218, 2023: 0.096842, 2024: 0.097889, 2025: 0.067308, 2026: 0.040950, 2027: 0.039339, 2028: 0.022984, 2029: 0.025251, 2030: 0.027289, 2031: 0.026986, @@ -2733,14 +2781,14 @@ def test_the_table_must_appear_once(): def test_the_october_2026_table_is_read_across_its_row(): """The real Table 1.13: the level under each year, the rates set by the - Government up until 2027, and the growth the hand-kept series held.""" + Government up until 2027, and each year's ratio as growth.""" growth = compute_level_growth(REAL_TABLE_1_13)[NLW_PATH] assert growth.line.levels == { year: float(text) for year, text in OCTOBER_2026_NLW.items() } assert growth.line.set_until == 2027 assert growth.line.table == "1.13" - assert growth.values == HAND_KEPT_NLW_GROWTH + assert growth.values == OCTOBER_2026_NLW_GROWTH assert infer_workbook_release(REAL_TABLE_1_13) == ("October", 2026) @@ -2754,6 +2802,132 @@ def test_october_2026_table_1_13_reproduces_the_committed_series(): assert rendered.encode() == committed +def projected_from_committed(workbook): + """The national_living_wage values the importer writes from ``workbook`` + into the committed yoy_growth.yaml, and the national living wage the + model projects from them after the £13.21 set for 2027.""" + rendered = render_level_series_yaml( + get_yoy_growth_path().read_text(), + compute_level_growth(workbook), + ("October", 2026), + ) + values = _values(rendered, NLW_PATH) + return values, model_wages(values, 2027, 13.21) + + +def test_an_estimate_a_millionth_above_half_a_penny_keeps_its_penny(): + """Table 1.13 with £14.625001 in 2031 (cell N5). Its ratio to 2030, + 0.027546, takes the model to £14.62, so the importer writes the growth + from the index's 2030 level to £14.625001, 0.027548, and the model + projects £14.63. Every other year keeps its ratio.""" + values, wages = projected_from_committed(edit_real_table_1_13({"N5": 14.625001})) + assert [wages[year] for year in range(2028, 2032)] == pytest.approx( + [13.51, 13.85, 14.23, 14.63], abs=1e-9 + ) + assert values[2031] == 0.027548 + assert {year: values[year] for year in range(2021, 2031)} == { + year: OCTOBER_2026_NLW_GROWTH[year] for year in range(2021, 2031) + } + + +def test_the_smallest_table_with_a_wrong_penny_keeps_its_penny(): + """£13.21 set for 2027 and an estimate of £13.515001 for 2028: the ratio, + 0.023089, takes the model to £13.51, so the importer writes 0.023092 and + the model projects £13.52.""" + workbook = nlw_workbook(nlw_rows((13.21, 13.515001), 2027, 2027)) + assert compute_level_growth(workbook)[NLW_PATH].values == {2028: 0.023089} + values, wages = projected_from_committed(workbook) + assert values[2028] == 0.023092 + assert wages[2028] == pytest.approx(13.52, abs=1e-9) + + +def test_a_level_no_six_decimal_growth_can_reach_is_refused(): + """With £5,000 set for 2024, one step of the five-decimal index moves the + projection by about 5p, so no growth projects £5,000.02 for 2025.""" + workbook = nlw_workbook(nlw_rows((5000.0, 5000.0, 5000.02), 2023, 2024)) + written = _values(YOY_GROWTH_FIXTURE, NLW_PATH) + kept = {date(year, 1, 1): value for year, value in written.items() if year < 2024} + for micros in range(-100, 101): + index = compound_model_index( + {**kept, date(2024, 1, 1): 0.0, date(2025, 1, 1): micros / 1_000_000} + ) + assert projected_rate_units(index, 2024, 5000.0, 2025, 0.01) != 500002 + with pytest.raises(ValueError, match="no 2025 growth to six decimals"): + render_nlw(workbook=workbook) + + +def test_a_table_needs_an_index_in_the_year_set(): + """Rates set only in the table's first year, with no earlier year kept: + the index starts a year later, so the model has no index level to + project the rate set from.""" + workbook = nlw_workbook(nlw_rows(NLW_LEVELS, 2016, 2016)) + with pytest.raises(ValueError, match="the index starts in 2017, after 2016"): + render_nlw(EARLY_EARNINGS_FIXTURE, workbook) + + +def test_a_table_past_the_model_indices_is_refused(): + workbook = nlw_workbook(nlw_rows(NLW_LEVELS, 2036, 2037)) + with pytest.raises( + ValueError, match="compounds its indices to 2039, so it cannot project the 2040" + ): + render_nlw(EARLY_EARNINGS_FIXTURE, workbook) + + +def test_the_series_rounds_as_the_rates_it_projects(): + """Every parameter the national living wage index uprates rounds each + projected rate to the nearest penny, the rounding the importer checks.""" + (spec,) = import_obr_forecasts.LEVEL_SERIES_SPECS + index = "gov.economic_assumptions.indices.obr.national_living_wage" + found = [] + + def visit(node): + if isinstance(node, dict): + uprating = (node.get("metadata") or {}).get("uprating") + if uprating == index: + found.append(None) + elif isinstance(uprating, dict) and uprating["parameter"] == index: + found.append(uprating.get("rounding")) + for child in node.values(): + visit(child) + elif isinstance(node, list): + for child in node: + visit(child) + + for path in (Path(__file__).parents[1] / "parameters").rglob("*.yaml"): + if index in path.read_text(): + visit(yaml.safe_load(path.read_text())) + assert len(found) == 6 + assert all( + rounding == {"type": "nearest", "interval": spec.rounding} for rounding in found + ) + + +@settings(max_examples=60, deadline=None, derandomize=True) +@given( + growth=st.dictionaries( + st.integers(2010, 2030), + st.integers(-200_000, 300_000).map(lambda micros: micros / 1_000_000), + min_size=1, + max_size=12, + ), + offset=st.integers(0, 8), + rate=st.floats(min_value=0.5, max_value=2000, allow_nan=False), +) +def test_the_importer_projects_rates_as_the_model_does(growth, offset, rate): + """Differential: compound_model_index equals the index + create_economic_assumption_indices builds from the same series, and + projected_rate_units, in pennies, equals the rate uprate_parameters + projects with the national living wage's metadata, in every year from + the one after the rate set to 2039.""" + set_until = min(growth) + offset + index = compound_model_index({date(year, 1, 1): v for year, v in growth.items()}) + assert index == model_index(growth) + wages = model_wages(growth, set_until, rate) + for year in range(set_until + 1, 2040): + units = projected_rate_units(index, set_until, rate, year, 0.01) + assert units * 0.01 == wages[year], year + + def edit_real_table_1_13(edits): """The real October 2026 workbook with cells of sheet 1.13 set: text as an inline string, a number as a number, None to empty the cell. It @@ -2842,6 +3016,23 @@ def edit_real_table_1_13(edits): {"B6": NLW_NOTE.format(set_until=2035, estimates_after=2035)}, "up until 2035, outside its years 2020-2031", ), + "set_before_the_table": ( + {"B6": NLW_NOTE.format(set_until=2019, estimates_after=2019)}, + "up until 2019, outside its years 2020-2031", + ), + "note_sets_two_years": ( + { + "B6": "The NLW has been set by the Government up until 2027. Figures " + "beyond 2027 are OBR estimates. The NLW has been set by the " + "Government up until 2028." + }, + r"names more than one year the Government has set the rate up until, " + r"\[2027, 2028\]", + ), + "estimates_beyond_another_year": ( + {"B7": "Figures beyond 2028 are OBR estimates."}, + r"and no others; it names \[2027, 2028\]", + ), } @@ -2855,6 +3046,23 @@ def test_a_changed_layout_of_the_real_table_is_refused(edits, message): compute_level_growth(edit_real_table_1_13(edits)) +@pytest.mark.parametrize( + "edits", + [ + {"B5": "National Living Wage (NLW)1"}, + {"B5": "National Living Wage (NLW)\u00b9"}, + {"B2": "1.13 National Living Wage\u00b2", "C3": "£ per hour\u00b3"}, + ], + ids=["label_digit", "label_superscript", "title_and_unit_superscript"], +) +def test_a_footnote_marker_is_read_past(edits): + """A footnote marker after the label's closing bracket, as a digit or a + superscript, or after the title or unit, leaves the table as it reads.""" + assert compute_level_growth(edit_real_table_1_13(edits)) == compute_level_growth( + REAL_TABLE_1_13 + ) + + def test_the_real_table_reads_the_same_with_no_cell_changed(): assert compute_level_growth(edit_real_table_1_13({})) == compute_level_growth( REAL_TABLE_1_13 @@ -2875,7 +3083,7 @@ def test_the_labelled_row_is_read_not_the_row_below_the_years(): } assert ( compute_level_growth(edit_real_table_1_13(edits))[NLW_PATH].values - == HAND_KEPT_NLW_GROWTH + == OCTOBER_2026_NLW_GROWTH ) @@ -3030,16 +3238,19 @@ def earnings_fixture(rate): first_year=st.integers(2015, 2027), data=st.data(), ) -def test_the_model_index_reproduces_the_published_levels( +def test_the_model_projects_the_published_levels_to_the_penny( first_level, changes, first_year, data ): """For any national living wage-like table (£5 to £20 in its first year, - up to 12 years, each changing by -5% to +10%): each year's growth is its - level over the year before's, to six decimals; the years up to the one - the note names fall under the rates set and the rest under the OBR's - estimates; and the index the model compounds from the written series - exceeds zero and carries the table's second-year level to within half a - penny of every later level.""" + up to 12 years, each changing by -5% to +10%): the importer reads each + year's growth as its level over the year before's, to six decimals; the + years up to the one the note names fall under the rates set and the rest + under the OBR's estimates; the index the model compounds from the + written series exceeds zero; and from the table's level in the year the + note names, the model projects every later level to the penny. The + importer writes another growth only for an estimate whose ratio, after + the years before as written, projects another penny. It refuses a table + whose index starts after the year the note names.""" levels = [first_level] for change in changes: levels.append(levels[-1] * (1 + change)) @@ -3049,19 +3260,36 @@ def test_the_model_index_reproduces_the_published_levels( workbook = nlw_workbook(nlw_rows(tuple(levels), first_year, set_until)) growth = compute_level_growth(workbook)[NLW_PATH] assert growth.line.levels == published - for year in years[1:]: - assert growth.values[year] == round( - published[year] / published[year - 1] - 1, 6 - ) - + ratios = { + year: round(published[year] / published[year - 1] - 1, 6) + 0.0 + for year in years[1:] + } + assert growth.values == ratios + + kept = _values(EARLY_EARNINGS_FIXTURE, NLW_PATH) + start = min([year for year in kept if year < years[1]] + [years[1]]) + if set_until < start: + event("refused: no index in the year set") + with pytest.raises(ValueError, match=f"the index starts in {start}"): + render_nlw(EARLY_EARNINGS_FIXTURE, workbook) + return rendered = render_nlw(EARLY_EARNINGS_FIXTURE, workbook) values = _values(rendered, NLW_PATH) - index = model_index(values) - assert all(level > 0 for level in index.values()) - base = years[1] + assert all(level > 0 for level in model_index(values).values()) + wages = model_wages(values, set_until, published[set_until]) + for year in years: + if year > set_until: + assert wages[year] == pytest.approx(round(published[year], 2), abs=1e-9) for year in years[1:]: - carried = published[base] * index[year] / index[base] - assert abs(carried - published[year]) < 0.005, year + if values[year] != ratios[year]: + event("an estimate's growth changed") + assert year > set_until + ratio_wages = model_wages( + {**values, year: ratios[year]}, set_until, published[set_until] + ) + assert ratio_wages[year] != pytest.approx( + round(published[year], 2), abs=1e-9 + ) block = rendered[rendered.index(" national_living_wage:") :] kind = None @@ -3093,6 +3321,36 @@ def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): render_nlw(content, workbook) +def test_a_finished_index_that_overflows_is_refused(): + """£0.00001 for 2027 and £1.7e303 for 2028 give finite growth, 1.7e308, + which the table's own index check, from 1 in 2027, passes. In the + committed file the index exceeds 1 by 2027, so the finished series' + index overflows in 2028: the finished-index check refuses that series, + and the importer refuses to write the table.""" + workbook = nlw_workbook(nlw_rows((1e-5, 1.7e303), 2027, 2027)) + growth = compute_level_growth(workbook)[NLW_PATH] + assert math.isfinite(growth.values[2028]) and growth.values[2028] > 1e308 + committed = get_yoy_growth_path().read_text() + finished = { + **{ + date(year, 1, 1): value + for year, value in _values(committed, NLW_PATH).items() + if year < 2028 + }, + date(2028, 1, 1): growth.values[2028], + **{ + date(year, 1, 1): value + for year, value in average_earnings_years(committed, 2028).items() + }, + } + with pytest.raises( + ValueError, match="national_living_wage: the index reaches inf in 2028" + ): + import_obr_forecasts.check_index_stays_positive(NLW_PATH, finished) + with pytest.raises(ValueError, match="national_living_wage: no 2028 growth"): + render_level_series_yaml(committed, {NLW_PATH: growth}, ("October", 2026)) + + def test_growth_past_every_float_is_refused(): with pytest.raises(ValueError, match="takes the index to inf"): compute_level_growth(nlw_workbook(nlw_rows((1e-300, 1e300), 2023, 2023))) @@ -3124,7 +3382,9 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): series, with the year kept before it and average_earnings (rising 2% or falling 51% a year) after it, reaches zero by the end of the growth file. Otherwise that index stays above zero in every year, and the - model's index agrees with it as far as the model compounds.""" + model's index agrees with it as far as the model compounds. The property + leaves the penny check out, as levels this far apart rarely round to a + penny the index can reach.""" content = earnings_fixture(earnings) workbook = nlw_workbook(nlw_rows(tuple(levels), 2023, 2023)) values = { @@ -3143,22 +3403,28 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): assert growth.values == values with pytest.MonkeyPatch.context() as patch: patch.setattr( - import_obr_forecasts, "check_index_stays_positive", lambda *_: None + import_obr_forecasts, + "penny_preserving_growth", + lambda _, series, __: series, ) - unchecked = render_nlw(content, workbook) - written = _values(unchecked, NLW_PATH) - finished = compounded(written, import_obr_forecasts.LAST_INDEX_YEAR) - # The model compounds the same index, as far as 2039. - model = model_index(written) - assert {year: finished[year] for year in model} == model - assert all(math.isfinite(level) for level in finished.values()) - try: - rendered = render_nlw(content, workbook) - except ValueError as error: - event("refused: finished series") - assert "national_living_wage: the index reaches" in str(error) - assert any(level <= 0 for level in finished.values()) - return + with pytest.MonkeyPatch.context() as unchecked_patch: + unchecked_patch.setattr( + import_obr_forecasts, "check_index_stays_positive", lambda *_: None + ) + unchecked = render_nlw(content, workbook) + written = _values(unchecked, NLW_PATH) + finished = compounded(written, import_obr_forecasts.LAST_INDEX_YEAR) + # The model compounds the same index, as far as 2039. + model = model_index(written) + assert {year: finished[year] for year in model} == model + assert all(math.isfinite(level) for level in finished.values()) + try: + rendered = render_nlw(content, workbook) + except ValueError as error: + event("refused: finished series") + assert "national_living_wage: the index reaches" in str(error) + assert any(level <= 0 for level in finished.values()) + return event("written") assert rendered == unchecked assert all(level > 0 for level in finished.values()) diff --git a/policyengine_uk/tests/test_minimum_wage_projection.py b/policyengine_uk/tests/test_minimum_wage_projection.py index daa251162e..675e779d29 100644 --- a/policyengine_uk/tests/test_minimum_wage_projection.py +++ b/policyengine_uk/tests/test_minimum_wage_projection.py @@ -107,8 +107,8 @@ def test_rates_are_the_last_rate_set_uprated_by_the_index(year): # OBR EFO October 2026, detailed forecast tables (economy), Table 1.13: the # national living wage in April of each year, £ an hour. "The NLW has been set # by the Government up until 2027 ... Figures beyond 2027 are OBR estimates -# consistent with the rest of our economy forecast." The table is also in -# tests/fixtures/obr_economy, which the importer reads. +# consistent with the rest of our economy forecast." tests/fixtures/obr_economy +# holds the same table, which the importer reads. OBR_NATIONAL_LIVING_WAGE = { 2020: 8.72, 2021: 8.91, diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 204be22210..e68b2e7e3d 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -6,7 +6,7 @@ import os import re import shutil -from dataclasses import dataclass +from dataclasses import dataclass, replace from datetime import date from decimal import Decimal from functools import lru_cache @@ -1471,16 +1471,17 @@ def receipts_references( def check_index_stays_positive(path: tuple[str, ...], values: dict) -> None: """Compound a rendered series as create_economic_assumption_indices does, from 1 in its first listed year, rounding to five decimals. - An index that rounds to zero would zero the base in every later year.""" + An index that rounds to zero would zero the base in every later year, and + one that overflows would make it infinite.""" growth = {str(instant): value for instant, value in values.items()} index = 1.0 for year in range(int(min(growth)[:4]) + 1, LAST_INDEX_YEAR + 1): in_force = max(instant for instant in growth if instant <= f"{year}-01-01") index = round(index * (1 + growth[in_force]), 5) - if index <= 0: + if not (math.isfinite(index) and index > 0): raise ValueError( f"{'.'.join(path)}: the index reaches {index} in {year} at five " - "decimals, so the base is too small to grow from" + "decimals, which the model cannot compound from" ) @@ -1572,21 +1573,25 @@ class LevelSeriesSpec: """An economy table that publishes a series as levels across a row, not as growth down a column. - ``path`` is the series' place under ``obr``. The importer finds the - table by its title, as it finds receipts tables. The table's header - declares one unit, ``unit``; one row of cells holds consecutive calendar - years; and one row below it, labelled ``row``, holds a level under each - year and nowhere else. The importer compares titles, units and labels - lowercased, without footnote markers. A note that the rate "has been set - by the Government up until" a year, and that figures beyond that year - "are OBR estimates", splits the years the Government has set from the - OBR's estimates. + ``path`` names the series under ``obr``. The importer finds the table by + its title, as it finds receipts tables. The table's header declares one + unit, ``unit``; one row of cells holds consecutive calendar years; and + one row below it, labelled ``row``, holds a level under each year and + nowhere else. The importer compares titles, units and labels lowercased, + without a footnote marker: digits or superscript digits after a letter or + a closing bracket, as in "National Living Wage (NLW)1". A note that the + rate "has been set by the Government up until" a year, and that figures + beyond that year "are OBR estimates", splits the years the Government has + set from the OBR's estimates; every such statement in the table must name + the same year. The parameters the series projects round each projected + rate to the nearest ``rounding``. """ path: tuple[str, ...] title: str unit: str row: str + rounding: float LEVEL_SERIES_SPECS = ( @@ -1595,8 +1600,14 @@ class LevelSeriesSpec: title="national living wage", unit="£ per hour", row="national living wage (nlw)", + # gov.hmrc.minimum_wage rounds each projected rate to the nearest penny. + rounding=0.01, ), ) +# A footnote marker ending a level table's title, unit or label. +LEVEL_FOOTNOTE_RE = re.compile( + r"(?<=[a-z)\]])(?:\d+|[\u2070\u00b9\u00b2\u00b3\u2074-\u2079]+)$", re.IGNORECASE +) SET_BY_GOVERNMENT_RE = re.compile(r"set by the government up until (\d{4})") OBR_ESTIMATES_RE = re.compile(r"figures beyond (\d{4}) are obr estimates") @@ -1614,12 +1625,20 @@ class LevelLine: @dataclass(frozen=True) class LevelGrowth: - """Growth in a level series, by year, rounded to six decimals: each - year's level over the year before's, from the table's second year to - its last.""" + """Growth in a level series, by year, to six decimals, from the table's + second year to its last. compute_level_growth writes each year's level + over the year before's; penny_preserving_growth then changes any OBR + estimate's growth that would project the wrong penny.""" values: dict[int, float] line: LevelLine + spec: LevelSeriesSpec + + +def level_label(value: str | None) -> str: + """A level table's title, unit or label as the importer compares it: + lowercased, with single spaces and without a footnote marker.""" + return LEVEL_FOOTNOTE_RE.sub("", normalise_label(value)) def _year_columns(row: dict[str, str | None]) -> dict[str, int]: @@ -1635,26 +1654,34 @@ def _year_columns(row: dict[str, str | None]) -> dict[str, int]: def _set_until(rows: list[dict[str, str | None]], where: str) -> int: - """The last year the Government has set, from the table's note.""" - notes = [ - normalise_label(value) - for row in rows - for value in row.values() - if SET_BY_GOVERNMENT_RE.search(normalise_label(value)) - ] + """The last year the Government has set, from the table's note. Every + statement in the table of the year the Government has set the rate up + until, or of the year beyond which figures are OBR estimates, must name + the same year.""" + texts = [normalise_label(value) for row in rows for value in row.values()] + notes = [text for text in texts if SET_BY_GOVERNMENT_RE.search(text)] if len(notes) != 1: raise ValueError( f"{where} needs one note saying which years the Government has set " f"and which are OBR estimates; found {len(notes)}" ) - set_until = SET_BY_GOVERNMENT_RE.search(notes[0]).group(1) - estimates = OBR_ESTIMATES_RE.search(notes[0]) - if estimates is None or estimates.group(1) != set_until: + set_years = sorted({int(year) for year in SET_BY_GOVERNMENT_RE.findall(notes[0])}) + if len(set_years) != 1: raise ValueError( - f"{where}'s note does not say the figures beyond {set_until} are " - f"OBR estimates: {notes[0]!r}" + f"{where}'s note names more than one year the Government has set " + f"the rate up until, {set_years}: {notes[0]!r}" ) - return int(set_until) + set_until = set_years[0] + estimates = sorted( + {int(year) for text in texts for year in OBR_ESTIMATES_RE.findall(text)} + ) + if estimates != [set_until]: + raise ValueError( + f"{where} does not say the figures beyond {set_until} are OBR " + f"estimates, and no others; it names {estimates or 'no year'}: " + f"{notes[0]!r}" + ) + return set_until def find_level_line( @@ -1672,7 +1699,7 @@ def find_level_line( (table, sheet, table_rows) for sheet, rows in sheets.items() for table, title, table_rows in sheet_tables(rows) - if title == spec.title + if level_label(title) == spec.title ] if len(tables) != 1: found = ", ".join(f"sheet {sheet}" for _, sheet, _ in tables) or "none" @@ -1696,7 +1723,7 @@ def find_level_line( ) if column_index(column) < first_year_column and normalise_label(value) ] - if labels and receipts_label(labels[0]) == spec.row: + if labels and level_label(labels[0]) == spec.row: matches.append((index, labels[0], row)) if len(matches) != 1: raise ValueError( @@ -1706,10 +1733,10 @@ def find_level_line( if index < header: raise ValueError(f'{where} has its "{spec.row}" row above its years') units = [ - normalise_label(value) + level_label(value) for header_row in rows[:header] for value in header_row.values() - if normalise_label(value) + if level_label(value) ] if units != [spec.unit]: raise ValueError( @@ -1748,7 +1775,7 @@ def find_level_line( table = sheet return LevelLine( table=table, - label=FOOTNOTE_RE.sub("", " ".join(str(label).split())), + label=LEVEL_FOOTNOTE_RE.sub("", " ".join(str(label).split())), levels=dict(sorted(levels.items())), set_until=set_until, ) @@ -1778,10 +1805,159 @@ def compute_level_growth( f"{index} at five decimals, which the model cannot " "compound from" ) - result[spec.path] = LevelGrowth(values, line) + result[spec.path] = LevelGrowth(values, line, spec) return result +# create_economic_assumption_indices compounds each index to 2039. +MODEL_INDEX_LAST_YEAR = 2039 + + +def compound_model_index(values: dict) -> dict[int, float]: + """The index create_economic_assumption_indices compounds from a growth + series: 1 in its first listed year, then each year's index times one plus + the growth in force on 1 January, rounded to five decimals, to 2039.""" + growth = {str(instant): value for instant, value in values.items()} + first = int(min(growth)[:4]) + index = {first: 1.0} + for year in range(first + 1, MODEL_INDEX_LAST_YEAR + 1): + in_force = max(instant for instant in growth if instant <= f"{year}-01-01") + index[year] = round(index[year - 1] * (1 + growth[in_force]), 5) + return index + + +def projected_rate_units( + index: dict[int, float], set_until: int, rate: float, year: int, rounding: float +) -> float: + """The rate uprate_parameters (policyengine-core) projects for ``year`` + from ``rate``, the last one set, in ``set_until``, as a whole number of + ``rounding``: the rate times the index's growth since, over ``rounding``, + rounded to the nearest whole number. Infinity if that overflows.""" + projected = rate * (index[year] / index[set_until]) / rounding + return round(projected) if math.isfinite(projected) else math.inf + + +def penny_preserving_growth( + path: tuple[str, ...], series: LevelGrowth, kept: dict +) -> LevelGrowth: + """``series`` with each OBR estimate's growth chosen so that the model + projects the table's level, as both round to the nearest + ``series.spec.rounding``. + + The model compounds ``kept``, the earlier years the file keeps, and then + ``series`` into an index (compound_model_index), and projects each rate + after the last one set from it (projected_rate_units). The importer takes + the table's level in the year the Government has set the rate up until + as that rate. Each later year keeps its growth if the projection rounds + to the table's level. Otherwise the importer takes the growth, to six + decimals, from the level the index carries the rate to in the year before + to the table's level, or else the nearest six-decimal growth to that + which does round to the table's level. It refuses a year that no + six-decimal growth can reproduce. + """ + name = ".".join(path) + line = series.line + rounding = series.spec.rounding + set_until, last = line.set_until, max(line.levels) + if last > MODEL_INDEX_LAST_YEAR: + raise ValueError( + f"{name}: the model compounds its indices to {MODEL_INDEX_LAST_YEAR}, " + f"so it cannot project the {last} level" + ) + values = dict(series.values) + growth = {str(instant): value for instant, value in kept.items()} + growth.update({f"{year}-01-01": value for year, value in values.items()}) + first = int(min(growth)[:4]) + if set_until < first: + raise ValueError( + f"{name}: the index starts in {first}, after {set_until}, the last " + "year the Government has set, so the model cannot project from it" + ) + rate = line.levels[set_until] + for year in range(set_until + 1, last + 1): + index = compound_model_index(growth) + for base in (set_until, year - 1): + if not (math.isfinite(index[base]) and index[base] > 0): + raise ValueError( + f"{name}: the index reaches {index[base]} in {base}, so the " + f"model cannot project the {year} level" + ) + refusal = ValueError( + f"{name}: no {year} growth to six decimals lets the model project " + f"the table's {year} level, {line.levels[year]!r}, from its " + f"{set_until} rate to the nearest {rounding}" + ) + if not math.isfinite(line.levels[year] / rounding): + raise refusal + target = round(line.levels[year] / rounding) + + def units(micros: int) -> float: + try: + value = micros / 1_000_000 + except OverflowError: + return math.inf + trial = compound_model_index({**growth, f"{year}-01-01": value}) + return projected_rate_units(trial, set_until, rate, year, rounding) + + written = round(Decimal(repr(values[year])) * 1_000_000) + if units(written) == target: + continue + + def first_above(bound: float) -> int: + """The least whole number of millionths, from -1 million (growth + of -100%), whose projection exceeds ``bound``. The projection + never falls as the growth rises, so a bisection finds it.""" + low, high = -1_000_000, max(written, 1) + if units(low) > bound: + return low + while units(high) <= bound: + low, high = high, 2 * high + while high - low > 1: + middle = (low + high) // 2 + if units(middle) > bound: + high = middle + else: + low = middle + return high + + lowest, highest = first_above(target - 1), first_above(target) - 1 + if lowest > highest: + raise refusal + carried = rate * (index[year - 1] / index[set_until]) + ratio = line.levels[year] / carried - 1 + sequential = round(ratio * 1e6) if math.isfinite(ratio * 1e6) else lowest + values[year] = min(max(sequential, lowest), highest) / 1_000_000 + 0.0 + growth[f"{year}-01-01"] = values[year] + return replace(series, values=values) + + +def _kept_level_values(obr: dict, path: tuple[str, ...], series: LevelGrowth) -> dict: + """The values a yoy_growth.yaml's ``obr`` mapping holds for a level + series before its table's second year, which a render keeps.""" + node = obr + for key in path: + node = node[key] + return { + instant: value + for instant, value in node["values"].items() + if int(str(instant)[:4]) < min(series.values) + } + + +def penny_preserving_level_growth( + content: str, growth: dict[tuple[str, ...], LevelGrowth] +) -> dict[tuple[str, ...], LevelGrowth]: + """penny_preserving_growth for each level series, given the years before + its table that ``content``, a yoy_growth.yaml, keeps.""" + obr = yaml.safe_load(content)["obr"] + return { + path: penny_preserving_growth( + path, series, _kept_level_values(obr, path, series) + ) + for path, series in growth.items() + } + + def average_earnings_years(content: str, after_year: int) -> dict[int, float]: """``average_earnings`` after ``after_year``, year for year: the path ``average_earnings_path`` gives, with an entry in every year @@ -1860,15 +2036,22 @@ def render_level_series_yaml( yoy_growth.yaml. It keeps the years before the table's second year, so the series' index - keeps its base year, and the series' other references. Years after the - table follow average_earnings as ``content`` holds it. It checks that - the finished series' index stays above zero, and that the result - differs from ``content`` only in those values and that reference. + keeps its base year, and the series' other references. It writes the + growth penny_preserving_growth chooses, so the model projects each of the + OBR's estimates to the table's penny. Years after the table follow + average_earnings as ``content`` holds it. It checks that the finished + series' index stays above zero, and that the result differs from + ``content`` only in those values and that reference. """ month, year = release expected = copy.deepcopy(yaml.safe_load(content)) lines = content.split("\n") for path, series in growth.items(): + node = expected["obr"] + for key in path: + node = node[key] + kept = _kept_level_values(expected["obr"], path, series) + series = penny_preserving_growth(path, series, kept) tail = average_earnings_years(content, max(series.values)) start, end = _series_block(lines, ("obr", *path)) indent = 2 * len(path) + 2 @@ -1906,14 +2089,6 @@ def render_level_series_yaml( lines[title] = f"{pad} - title: {reference['title']}" lines[title + 1] = f"{pad} href: {reference['href']}" - node = expected["obr"] - for key in path: - node = node[key] - kept = { - instant: value - for instant, value in node["values"].items() - if int(str(instant)[:4]) < min(series.values) - } node["values"] = { **kept, **{ @@ -2143,8 +2318,14 @@ def print_receipts_summary(growth: dict[tuple[str, ...], ReceiptsGrowth]) -> Non def print_level_summary(growth: dict[tuple[str, ...], LevelGrowth]) -> None: print(f"Updating {len(growth)} OBR level series") for path, series in growth.items(): + levels = series.line.levels window = ", ".join( f"{year}: {format_growth(value)}" + + ( + "" + if value == round(levels[year] / levels[year - 1] - 1, 6) + 0.0 + else " (not the levels' ratio, which projects another penny)" + ) for year, value in sorted(series.values.items()) ) print( @@ -2421,7 +2602,6 @@ def main(argv: list[str] | None = None) -> int: level_growth = None if not args.gaps_only and not args.skip_national_living_wage: level_growth = compute_level_growth(workbook_bytes) - print_level_summary(level_growth) calendar_values = ( read_calendar_values(source_yaml_path) if args.gaps_only else series_values ) @@ -2463,6 +2643,11 @@ def main(argv: list[str] | None = None) -> int: ) if level_growth is not None: yoy_path = args.output or source_yaml_path + # The growth that keeps each penny depends on the years the file keeps + # before the table, so the summary waits for the file. + print_level_summary( + penny_preserving_level_growth(outputs[yoy_path], level_growth) + ) outputs[yoy_path] = render_level_series_yaml( outputs[yoy_path], level_growth, (month, year) ) From fad8832f35566f44c710dc40f337cb35d2dc6383 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 12:11:10 -0400 Subject: [PATCH 08/22] Draw the penny property's year set from years the index covers MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The property now always reaches the penny check: the note names a year the series' index covers, and an explicit example, £13.525001 for 2028 after £13.21 set for 2027, makes the importer change an estimate's growth. test_a_table_needs_an_index_in_the_year_set covers the other years. The economic assumptions README states the penny rule and the new refusals. Co-Authored-By: Claude Opus 5.5 --- .../gov/economic_assumptions/README.md | 15 +++++++---- .../tests/test_import_obr_forecasts.py | 27 ++++++++++--------- 2 files changed, 25 insertions(+), 17 deletions(-) diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index 819321a252..8a40a060d9 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -71,11 +71,16 @@ series. It finds the table by its title, reads the level under each year across the table's one row of years, and writes each year's level over the year before's, to six decimals. The table's note says up to which year the Government has set the rate, and the importer writes those years and the -OBR's estimates under separate comments. The importer stops before writing -any file if the table declares a unit other than £ per hour, or lacks one row -of years and one row labelled "National Living Wage (NLW)". It also stops if -the years skip or repeat, if a year lacks a positive level, if a figure has no -year above it, or if the table lacks that note. `test_minimum_wage_projection.py` +OBR's estimates under separate comments. It then projects each estimate as +the model does, from the last rate set through the index to the nearest +penny. Where a year's ratio would leave the model a penny off the table, it +writes the six-decimal growth that keeps the table's penny. The importer stops +before writing any file if the table declares a unit other than £ per hour, +or lacks one row of years and one row labelled "National Living Wage (NLW)". +It also stops if the years skip or repeat, if a year lacks a positive level, +if a figure has no year above it, if the table lacks that note or its notes +name different years, or if no six-decimal growth keeps an estimate's penny. +`test_minimum_wage_projection.py` holds the October 2026 table, checks it against the importer's fixture, and fails if the series stops reproducing the table to the penny, or if its long-run years stop matching `average_earnings`. diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index eb757acc25..7b692edf12 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3236,10 +3236,17 @@ def earnings_fixture(rate): first_level=st.floats(min_value=5, max_value=20, allow_nan=False), changes=st.lists(NLW_CHANGES, min_size=1, max_size=11), first_year=st.integers(2015, 2027), - data=st.data(), + set_offset=st.integers(0, 11), +) +# £13.525001 for 2028: its ratio to the £13.21 set for 2027 projects £13.52. +@example( + first_level=13.21, + changes=[0.023845647236941714], + first_year=2027, + set_offset=0, ) def test_the_model_projects_the_published_levels_to_the_penny( - first_level, changes, first_year, data + first_level, changes, first_year, set_offset ): """For any national living wage-like table (£5 to £20 in its first year, up to 12 years, each changing by -5% to +10%): the importer reads each @@ -3249,14 +3256,17 @@ def test_the_model_projects_the_published_levels_to_the_penny( written series exceeds zero; and from the table's level in the year the note names, the model projects every later level to the penny. The importer writes another growth only for an estimate whose ratio, after - the years before as written, projects another penny. It refuses a table - whose index starts after the year the note names.""" + the years before as written, projects another penny. The note names a + year the series' index covers (test_a_table_needs_an_index_in_the_year_set + covers the others).""" levels = [first_level] for change in changes: levels.append(levels[-1] * (1 + change)) years = range(first_year, first_year + len(levels)) published = dict(zip(years, levels)) - set_until = data.draw(st.integers(first_year, years[-1])) + kept = _values(EARLY_EARNINGS_FIXTURE, NLW_PATH) + start = min([year for year in kept if year < years[1]] + [years[1]]) + set_until = min(max(first_year, start) + set_offset, years[-1]) workbook = nlw_workbook(nlw_rows(tuple(levels), first_year, set_until)) growth = compute_level_growth(workbook)[NLW_PATH] assert growth.line.levels == published @@ -3266,13 +3276,6 @@ def test_the_model_projects_the_published_levels_to_the_penny( } assert growth.values == ratios - kept = _values(EARLY_EARNINGS_FIXTURE, NLW_PATH) - start = min([year for year in kept if year < years[1]] + [years[1]]) - if set_until < start: - event("refused: no index in the year set") - with pytest.raises(ValueError, match=f"the index starts in {start}"): - render_nlw(EARLY_EARNINGS_FIXTURE, workbook) - return rendered = render_nlw(EARLY_EARNINGS_FIXTURE, workbook) values = _values(rendered, NLW_PATH) assert all(level > 0 for level in model_index(values).values()) From 1b52679c05c7b4c1edc9506a8ee3fb3d5055a4d6 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 13:08:51 -0400 Subject: [PATCH 09/22] Test the summary's note on a growth that keeps the penny Co-Authored-By: Claude Opus 5.5 --- policyengine_uk/tests/test_import_obr_forecasts.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 7b692edf12..eaebcf516f 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3199,6 +3199,20 @@ def test_main_writes_nothing_without_a_readable_table(tmp_path, workbook, messag assert {path: path.read_text() for path in before} == before +def test_the_summary_marks_a_growth_that_keeps_the_penny(capsys): + """The summary main prints marks a year whose growth differs from its + levels' ratio: £13.525001 for 2028 after £13.21 set for 2027.""" + workbook = nlw_workbook(nlw_rows((13.21, 13.525001000000001), 2027, 2027)) + growth = import_obr_forecasts.penny_preserving_level_growth( + EARLY_EARNINGS_FIXTURE, compute_level_growth(workbook) + ) + import_obr_forecasts.print_level_summary(growth) + assert ( + "2028: 0.023848 (not the levels' ratio, which projects another penny)" + in capsys.readouterr().out + ) + + def test_main_can_leave_the_national_living_wage_alone(tmp_path): yoy, _ = write_tree(tmp_path) no_table = rebuild_workbook(make_test_xlsx(), drop={"1.13"}) From a88c4e4da36ea206d4afef1b6e138927396a35c7 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 13:50:46 -0400 Subject: [PATCH 10/22] Add earnings years only to average_earnings in the tail-years test The test added 2032 and 2033 after every "2031-01-01: 0.0200" line, and #2761's per-head series, indented two spaces deeper, hold that line too, so the edited fixture stopped parsing. It now edits average_earnings alone, as the test means to. Co-Authored-By: Claude Opus 5.5 --- policyengine_uk/tests/test_import_obr_forecasts.py | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index a4cfb82a5a..dcf2fb5ac0 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3330,10 +3330,16 @@ def test_render_writes_set_rates_estimates_then_average_earnings(): def test_the_series_lists_every_average_earnings_year_after_the_table(): + flat = FLAT_SERIES_FIXTURE.format(key="average_earnings") content = YOY_GROWTH_FIXTURE.replace( - " 2031-01-01: 0.0200\n", - " 2031-01-01: 0.0200\n 2032-01-01: 0.0200\n 2033-01-01: 0.0300\n", + flat, + flat.replace( + " 2031-01-01: 0.0200\n", + " 2031-01-01: 0.0200\n 2032-01-01: 0.0200\n" + " 2033-01-01: 0.0300\n", + ), ) + assert content != YOY_GROWTH_FIXTURE values = _values(render_nlw(content), NLW_PATH) earnings = _values(content, ("average_earnings",)) assert {y for y in values if y > 2027} == {2028} | {y for y in earnings if y > 2027} From 2823883d8669b0d18109dd08158cceb38d26f978 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 15:34:26 -0400 Subject: [PATCH 11/22] Test that an estimate takes the nearest growth that keeps its penny MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit £13.614999 for 2028 after £13.21 set for 2027: the ratio projects £13.62, every growth from 0.029904 to 0.030658 keeps £13.61, and the importer writes 0.030658, the one nearest the growth to the table's level. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index dcf2fb5ac0..01c6707bed 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3041,6 +3041,26 @@ def test_the_smallest_table_with_a_wrong_penny_keeps_its_penny(): assert wages[2028] == pytest.approx(13.52, abs=1e-9) +def test_an_estimate_just_below_half_a_penny_takes_the_nearest_growth(): + """£13.21 set for 2027 and an estimate of £13.614999 for 2028: the ratio, + 0.030659, takes the model to £13.62. Every growth from 0.029904 to 0.030658 + keeps £13.61, and the importer writes the one nearest the growth from the + £13.21 rate to £13.614999, 0.030658.""" + workbook = nlw_workbook(nlw_rows((13.21, 13.614999), 2027, 2027)) + assert compute_level_growth(workbook)[NLW_PATH].values == {2028: 0.030659} + values, wages = projected_from_committed(workbook) + for growth, wage in ( + (0.029903, 13.60), + (0.029904, 13.61), + (0.030658, 13.61), + (0.030659, 13.62), + ): + projected = model_wages({**values, 2028: growth}, 2027, 13.21)[2028] + assert projected == pytest.approx(wage, abs=1e-9), growth + assert values[2028] == 0.030658 + assert wages[2028] == pytest.approx(13.61, abs=1e-9) + + def test_a_level_no_six_decimal_growth_can_reach_is_refused(): """With £5,000 set for 2024, one step of the five-decimal index moves the projection by about 5p, so no growth projects £5,000.02 for 2025.""" From a473bf2eab44e2faeb3a2f1a7aac2e867422eaab Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 17:51:34 -0400 Subject: [PATCH 12/22] Project the national living wage from its parameter file's rate MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The importer took the rate it projects each OBR estimate from out of Table 1.13, unrounded, in the year the OBR's note names. The model projects from the last rate in gov.hmrc.minimum_wage.non_apprentice, so a table holding £13.210001 for 2027 passed growth that takes the model's £13.21 to £13.59, not the table's £13.60. - read_level_anchor reads the national living wage's last dated rate and its uprating metadata from non_apprentice.yaml. The penny step checks every table year after that rate's year, and refuses a table whose level in that year differs from the rate at the penny. The note's year labels the comments only. - build_model_index runs the model's generation steps, through create_economic_assumption_indices, on the statutory parameters with the rendered growth series, and the render refuses a table the model's index does not reach, or an index compound_model_index does not reproduce. MODEL_INDEX_LAST_YEAR goes. - check_level_index compounds the file's kept years before the table, as the model does, so a fall the kept years hold above zero passes. - The note's year must be one affirmative calendar year: the importer refuses "has not been set", a range such as 2027-28 and a five-digit year. - The generated comments name who acts: the government sets the rates, the OBR estimates them, the importer copies average_earnings. Co-Authored-By: Claude Opus 5.5 --- .../gov/economic_assumptions/yoy_growth.yaml | 6 +- .../tests/test_import_obr_forecasts.py | 644 ++++++++++++++---- policyengine_uk/utils/import_obr_forecasts.py | 446 ++++++++---- 3 files changed, 827 insertions(+), 269 deletions(-) diff --git a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml index a60fd9af08..67b580dab5 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml +++ b/policyengine_uk/parameters/gov/economic_assumptions/yoy_growth.yaml @@ -223,7 +223,7 @@ obr: # earnings "should continue to be the key reference point for future # rates". values: - # Set by the Government (2021-2027) from OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)": £8.72 in 2020 to £13.21 in 2027 + # The government set the rates for 2021-2027: £8.72 in 2020 to £13.21 in 2027 (OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)") 2021-01-01: 0.021789 2022-01-01: 0.066218 2023-01-01: 0.096842 @@ -231,12 +231,12 @@ obr: 2025-01-01: 0.067308 2026-01-01: 0.04095 2027-01-01: 0.039339 - # OBR estimates (2028-2031) from OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)": £13.21 in 2027 to £14.62 in 2031 + # The OBR estimates the rates for 2028-2031: £13.21 in 2027 to £14.62 in 2031 (OBR EFO October 2026, economy Table 1.13 "National Living Wage (NLW)") 2028-01-01: 0.022984 2029-01-01: 0.025251 2030-01-01: 0.027289 2031-01-01: 0.026986 - # Beyond the OBR forecast: the average_earnings long-run path above, year for year + # After the OBR forecast, the importer copies average_earnings year for year 2032-01-01: 0.0285 2033-01-01: 0.0309 2034-01-01: 0.0334 diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 01c6707bed..b5c1357443 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -2,6 +2,7 @@ import os from dataclasses import replace from datetime import date +from functools import partial from pathlib import Path from io import BytesIO from textwrap import indent @@ -12,7 +13,7 @@ import yaml from hypothesis import event, example, given, settings from hypothesis import strategies as st -from policyengine_core.parameters import ParameterNode +from policyengine_core.parameters import ParameterNode, config from policyengine_core.parameters.operations.uprate_parameters import ( uprate_parameters, ) @@ -20,6 +21,8 @@ from policyengine_uk.parameters.gov.economic_assumptions.create_economic_assumption_indices import ( create_economic_assumption_indices, ) +from policyengine_uk.system import system +from policyengine_uk.tax_benefit_system import statutory_parameters from policyengine_uk.utils import import_obr_forecasts from policyengine_uk.utils.import_obr_forecasts import ( @@ -28,11 +31,13 @@ STATUTORY_GAP_SPECS, _sheet_paths, ForecastGap, + LevelAnchor, MACHINE_GAMES_HIGHER_STAKE_SHARE, ReceiptsSeriesSpec, machine_games_spec, average_earnings_path, average_earnings_years, + build_model_index, compute_level_growth, compute_receipts_growth, compound_model_index, @@ -44,6 +49,7 @@ main, projected_rate_units, read_calendar_values, + read_level_anchor, read_workbook_sheets, render_forecast_gap_yaml, render_level_series_yaml, @@ -905,24 +911,33 @@ def run_main( receipts_path.write_bytes(receipts or make_receipts_xlsx()) previous_path = tmp_path / "previous_receipts.xlsx" previous_path.write_bytes(previous_receipts or make_previous_receipts_xlsx()) - return main( - [ - "--file", - str(economy), - "--forecast-start-year", - "2025", - "--forecast-years", - "2", - "--release-month", - "March", - "--release-year", - "2026", - "--receipts-file", - str(receipts_path), - *(["--previous-receipts-file", str(previous_path)] if previous else []), - *extra, - ] - ) + # The model's generation steps cannot read the fixture's short growth + # series (its rpi starts in 2024), so main takes the national living + # wage's index from create_economic_assumption_indices on the fixture + # alone, and its anchor from the rate the test table's note sets. + with pytest.MonkeyPatch.context() as patch: + patch.setattr(import_obr_forecasts, "build_model_index", fixture_index) + patch.setattr( + import_obr_forecasts, "read_level_anchor", lambda spec: CLI_NLW_ANCHOR + ) + return main( + [ + "--file", + str(economy), + "--forecast-start-year", + "2025", + "--forecast-years", + "2", + "--release-month", + "March", + "--release-year", + "2026", + "--receipts-file", + str(receipts_path), + *(["--previous-receipts-file", str(previous_path)] if previous else []), + *extra, + ] + ) def test_main_writes_growth_and_gaps_next_to_the_yaml_path(tmp_path): @@ -2788,9 +2803,9 @@ def test_a_written_series_keeps_every_index_positive(levels, schedules): assert index > 0 -# The national living wage (#2721): economy Table 1.13 publishes its levels, -# £ an hour, across a row of calendar years, and the importer writes their -# growth. +# The national living wage (#2721): the OBR publishes its levels, £ an hour, +# across a row of calendar years in economy Table 1.13, and the importer +# writes their growth. NLW_PATH = ("national_living_wage",) NLW_LEVELS = (10.0, 10.5, 11.0, 11.55, 12.0) @@ -2840,10 +2855,10 @@ def nlw_workbook(rows=None) -> bytes: return make_workbook({"1.13": nlw_sheet(nlw_rows() if rows is None else rows)}) -def model_index(values): - """The index the model compounds from a growth series - (create_economic_assumption_indices on a tree holding only it).""" - tree = ParameterNode( +def nlw_tree(values, **extra): + """A parameter tree holding only the national living wage's growth + series, ``values`` by year, and ``extra``.""" + return ParameterNode( "", data={ "gov": { @@ -2859,12 +2874,49 @@ def model_index(values): } } } + }, + **extra, + }, + ) + + +def index_years(tree, path=NLW_PATH): + """The index create_economic_assumption_indices wrote for ``obr.`` + in ``tree``, by year, in every year it wrote.""" + index = tree.get_child("gov.economic_assumptions.indices.obr." + ".".join(path)) + return { + int(entry.instant_str[:4]): entry.value + for entry in sorted(index.values_list, key=lambda entry: entry.instant_str) + } + + +def model_index(values): + """The index the model compounds from a growth series + (create_economic_assumption_indices on a tree holding only it), in + every year the builder writes.""" + tree = nlw_tree(values) + create_economic_assumption_indices(tree) + return index_years(tree) + + +def fixture_index(content, path): + """The index create_economic_assumption_indices builds from ``content``'s + ``obr.`` on a tree holding only ``content``'s growth series, as + policyengine-core loads them. The model's other generation steps read + series the test files hold too briefly, so renders of a test file take + their index from here.""" + tree = ParameterNode( + "", + data={ + "gov": { + "economic_assumptions": { + "yoy_growth": yaml.load(content, Loader=config.Loader) + } } }, ) create_economic_assumption_indices(tree) - index = tree.gov.economic_assumptions.indices.obr.national_living_wage - return {year: index(f"{year}-01-01") for year in range(min(values), 2040)} + return index_years(tree, path) MINIMUM_WAGE_DIR = Path(__file__).parents[1] / "parameters/gov/hmrc/minimum_wage" @@ -2873,25 +2925,26 @@ def model_index(values): NLW_UPRATING = yaml.safe_load((MINIMUM_WAGE_DIR / "non_apprentice.yaml").read_text())[ "brackets" ][-1]["amount"]["metadata"]["uprating"] +(NLW_SPEC,) = import_obr_forecasts.LEVEL_SERIES_SPECS +# The rate the CLI tests' Table 1.13 sets (nlw_rows: £11.00 up until 2025). +CLI_NLW_ANCHOR = LevelAnchor(date(2025, 4, 1), 11.0, NLW_UPRATING) -def model_wages(values, set_until, rate): - """The national living wage the model projects in April of each year - after ``set_until`` to 2039, from ``rate`` set in April ``set_until``: the - growth series through create_economic_assumption_indices, then a rate - with the national living wage's uprating metadata through - policyengine-core's uprate_parameters.""" +def uprated_wages(index, anchor): + """The rate policyengine-core's uprate_parameters projects in April of + each year after ``anchor``'s to the end of ``index``, from the anchor's + rate and with its uprating metadata.""" tree = ParameterNode( "", data={ "gov": { "economic_assumptions": { - "yoy_growth": { + "indices": { "obr": { "national_living_wage": { "values": { f"{year}-01-01": value - for year, value in values.items() + for year, value in index.items() } } } @@ -2899,14 +2952,50 @@ def model_wages(values, set_until, rate): } }, "wage": { - "metadata": {"uprating": NLW_UPRATING}, - "values": {f"{set_until}-04-01": rate}, + "metadata": {"uprating": anchor.uprating}, + "values": {anchor.instant.isoformat(): anchor.rate}, }, }, ) - create_economic_assumption_indices(tree) uprate_parameters(tree) - return {year: tree.wage(f"{year}-04-01") for year in range(set_until + 1, 2040)} + return { + year: tree.wage(f"{year}-04-01") + for year in range(anchor.instant.year + 1, max(index) + 1) + } + + +def model_wages(values, set_until, rate): + """The national living wage the model projects in April of each year + after ``set_until`` to the end of its index, from ``rate`` set in April + ``set_until``: the growth series through create_economic_assumption_indices + on a tree holding only it, then a rate with the national living wage's + uprating metadata through policyengine-core's uprate_parameters.""" + return uprated_wages( + model_index(values), LevelAnchor(date(set_until, 4, 1), rate, NLW_UPRATING) + ) + + +def wages_in_the_model(content): + """The national living wage the model projects in April of each year + after the rate the parameter file sets last, when ``content`` is its + yoy_growth.yaml: the index the model's own generation steps build + (build_model_index), carried from the file's rate by policyengine-core's + uprate_parameters.""" + return uprated_wages( + build_model_index(content, NLW_PATH), read_level_anchor(NLW_SPEC) + ) + + +def table_anchor(workbook): + """The rate a parameter file would hold for a test table: the table's + level in the year its note names, to the nearest penny, from April that + year.""" + line = compute_level_growth(workbook)[NLW_PATH].line + return LevelAnchor( + date(line.set_until, 4, 1), + round(line.levels[line.set_until] / 0.01) / 100, + NLW_UPRATING, + ) def compounded(values, last_year): @@ -3005,14 +3094,14 @@ def test_october_2026_table_1_13_reproduces_the_committed_series(): def projected_from_committed(workbook): """The national_living_wage values the importer writes from ``workbook`` into the committed yoy_growth.yaml, and the national living wage the - model projects from them after the £13.21 set for 2027.""" + model projects from them after the £13.21 the parameter file sets for + 2027.""" rendered = render_level_series_yaml( get_yoy_growth_path().read_text(), compute_level_growth(workbook), ("October", 2026), ) - values = _values(rendered, NLW_PATH) - return values, model_wages(values, 2027, 13.21) + return _values(rendered, NLW_PATH), wages_in_the_model(rendered) def test_an_estimate_a_millionth_above_half_a_penny_keeps_its_penny(): @@ -3061,6 +3150,138 @@ def test_an_estimate_just_below_half_a_penny_takes_the_nearest_growth(): assert wages[2028] == pytest.approx(13.61, abs=1e-9) +def test_the_anchor_is_the_last_national_living_wage_rate_in_its_file(): + """The importer projects from the rate the model projects from: the last + dated value of the national living wage, the last bracket of + gov.hmrc.minimum_wage.non_apprentice, as the model loads it, with that + amount's uprating metadata.""" + anchor = read_level_anchor(NLW_SPEC) + assert anchor == LevelAnchor(date(2027, 4, 1), 13.21, NLW_UPRATING) + amount = ( + statutory_parameters().gov.hmrc.minimum_wage.non_apprentice.brackets[-1].amount + ) + assert amount.values_list[0].instant_str == anchor.instant.isoformat() + assert amount.values_list[0].value == anchor.rate + + +def test_a_table_level_the_file_rounds_to_its_rate_projects_from_the_file(): + """£13.210001 for 2027, the year the parameter file sets £13.21, and an + estimate of £13.595001 for 2028. From the table's unrounded 2027 level + the ratio, 0.029145, would reach £13.60; the model projects from the + file's £13.21, where that growth reaches £13.59. The importer writes + 0.029148, and the model projects £13.60.""" + workbook = nlw_workbook(nlw_rows((13.210001, 13.595001), 2027, 2027)) + assert compute_level_growth(workbook)[NLW_PATH].values == {2028: 0.029145} + values, wages = projected_from_committed(workbook) + assert model_wages({**values, 2028: 0.029145}, 2027, 13.21)[2028] == ( + pytest.approx(13.59, abs=1e-9) + ) + assert values[2028] == 0.029148 + assert wages[2028] == pytest.approx(13.60, abs=1e-9) + + +def test_a_table_that_disagrees_with_the_rate_set_is_refused(): + """The real Table 1.13 with £13.22 for 2027, the year the parameter file + sets £13.21: the OBR and the file disagree about a rate the government + has set, so the importer refuses the table and names both figures.""" + with pytest.raises( + ValueError, + match=r"the OBR's 2027 level, 13\.22, differs to the nearest 0\.01 from " + r"13\.21, the rate gov\.hmrc\.minimum_wage\.non_apprentice holds from " + r"2027-04-01", + ): + projected_from_committed(edit_real_table_1_13({"J5": 13.22})) + + +def nlw_comments(rendered): + """The national_living_wage series' comment lines in a rendered file.""" + block = rendered[rendered.index(" national_living_wage:") :] + block = block[: block.index(" metadata:")] + return [line.strip() for line in block.split("\n") if line.strip().startswith("#")] + + +def test_a_note_a_year_after_the_files_rate_leaves_the_file_to_anchor(): + """The real table with a note that the government has set the rate up + until 2028, a year after the parameter file's last rate, and £13.515001 + for 2028. The model still projects 2028 from the file's £13.21 for 2027, + so the importer checks 2028 as well: it writes 0.023092, and the model + projects £13.52. The note still labels 2028 among the rates the + government set.""" + note = NLW_NOTE.format(set_until=2028, estimates_after=2028) + workbook = edit_real_table_1_13({"B6": note, "K5": 13.515001}) + values, wages = projected_from_committed(workbook) + assert values[2028] == 0.023092 + assert [wages[year] for year in range(2028, 2032)] == pytest.approx( + [13.52, 13.85, 14.23, 14.62], abs=1e-9 + ) + rendered = render_level_series_yaml( + get_yoy_growth_path().read_text(), + compute_level_growth(workbook), + ("October", 2026), + ) + assert nlw_comments(rendered)[-3:-1] == [ + "# The government set the rates for 2021-2028: £8.72 in 2020 to £13.52 " + 'in 2028 (OBR EFO October 2026, economy Table 1.13 "National Living ' + 'Wage (NLW)")', + "# The OBR estimates the rates for 2029-2031: £13.52 in 2028 to £14.62 " + 'in 2031 (OBR EFO October 2026, economy Table 1.13 "National Living ' + 'Wage (NLW)")', + ] + + +def test_a_note_a_year_before_the_files_rate_leaves_the_file_to_anchor(): + """The real table with a note that the government has set the rate up + until 2026, a year before the parameter file's last rate, and + £14.625001 for 2031. From the table's £12.71 for 2026 the ratio, + 0.027546, would reach £14.63; the model projects from the file's £13.21 + for 2027, where it reaches £14.62. The importer writes 0.027548, and the + model projects £14.63. The note still labels 2027 among the OBR's + estimates.""" + note = NLW_NOTE.format(set_until=2026, estimates_after=2026) + workbook = edit_real_table_1_13({"B6": note, "N5": 14.625001}) + values, wages = projected_from_committed(workbook) + assert {year: values[year] for year in range(2021, 2031)} == { + year: OCTOBER_2026_NLW_GROWTH[year] for year in range(2021, 2031) + } + assert values[2031] == 0.027548 + assert [wages[year] for year in range(2028, 2032)] == pytest.approx( + [13.51, 13.85, 14.23, 14.63], abs=1e-9 + ) + rendered = render_level_series_yaml( + get_yoy_growth_path().read_text(), + compute_level_growth(workbook), + ("October", 2026), + ) + assert nlw_comments(rendered)[-2].startswith( + "# The OBR estimates the rates for 2027-2031: £12.71 in 2026 to £14.63 in 2031" + ) + + +def test_a_fall_the_kept_years_hold_above_zero_is_written(): + """£13.21 for 2027 and £0.00005284 for 2028: growth of -0.999996. From 1 + in 2027 the five-decimal index would fall to 0.0, but the model's index + starts in 2021, and the years the file keeps take it to 1.48261 by 2027, + so it falls only to 0.00001. The importer writes the table, and the + model projects £0.00 for 2028, the table's penny.""" + workbook = nlw_workbook(nlw_rows((13.21, 0.00005284), 2027, 2027)) + values, wages = projected_from_committed(workbook) + assert values[2028] == -0.999996 + index = model_index(values) + assert (index[2021], index[2027], index[2028]) == (1.0, 1.48261, 0.00001) + assert wages[2028] == 0.0 + + +def test_a_level_fall_that_takes_the_index_to_zero_is_refused(): + """£13.21 for 2027 and £0.000000001 for 2028: the growth rounds to -1, + so the index the model compounds through the kept years reaches 0.0.""" + workbook = nlw_workbook(nlw_rows((13.21, 1e-9), 2027, 2027)) + with pytest.raises( + ValueError, + match="national_living_wage: 2028 growth of -1.0 takes the index to 0.0", + ): + projected_from_committed(workbook) + + def test_a_level_no_six_decimal_growth_can_reach_is_refused(): """With £5,000 set for 2024, one step of the five-decimal index moves the projection by about 5p, so no growth projects £5,000.02 for 2025.""" @@ -3069,7 +3290,8 @@ def test_a_level_no_six_decimal_growth_can_reach_is_refused(): kept = {date(year, 1, 1): value for year, value in written.items() if year < 2024} for micros in range(-100, 101): index = compound_model_index( - {**kept, date(2024, 1, 1): 0.0, date(2025, 1, 1): micros / 1_000_000} + {**kept, date(2024, 1, 1): 0.0, date(2025, 1, 1): micros / 1_000_000}, + 2025, ) assert projected_rate_units(index, 2024, 5000.0, 2025, 0.01) != 500002 with pytest.raises(ValueError, match="no 2025 growth to six decimals"): @@ -3077,20 +3299,42 @@ def test_a_level_no_six_decimal_growth_can_reach_is_refused(): def test_a_table_needs_an_index_in_the_year_set(): - """Rates set only in the table's first year, with no earlier year kept: + """A rate set only in the table's first year, with no earlier year kept: the index starts a year later, so the model has no index level to - project the rate set from.""" + project that rate from.""" workbook = nlw_workbook(nlw_rows(NLW_LEVELS, 2016, 2016)) with pytest.raises(ValueError, match="the index starts in 2017, after 2016"): render_nlw(EARLY_EARNINGS_FIXTURE, workbook) -def test_a_table_past_the_model_indices_is_refused(): +def test_a_table_renders_as_far_as_the_model_builds_the_index(): + """A table running to 2040, rendered into the committed file: where the + model's own builder ends the index before 2040, the importer refuses the + table; otherwise it writes it, and the model projects each estimate + after the £10.50 set for 2037 to the table's penny.""" workbook = nlw_workbook(nlw_rows(NLW_LEVELS, 2036, 2037)) - with pytest.raises( - ValueError, match="compounds its indices to 2039, so it cannot project the 2040" - ): - render_nlw(EARLY_EARNINGS_FIXTURE, workbook) + committed = get_yoy_growth_path().read_text() + anchor = LevelAnchor(date(2037, 4, 1), 10.5, NLW_UPRATING) + horizon = max(build_model_index(committed, NLW_PATH)) + render = partial( + render_level_series_yaml, + committed, + compute_level_growth(workbook), + ("October", 2026), + {NLW_PATH: anchor}, + ) + if horizon < 2040: + with pytest.raises( + ValueError, + match=f"the model's index ends in {horizon}, so it cannot project the 2040", + ): + render() + return + rendered = render() + wages = uprated_wages(build_model_index(rendered, NLW_PATH), anchor) + assert [wages[year] for year in (2038, 2039, 2040)] == pytest.approx( + [11.0, 11.55, 12.0], abs=1e-9 + ) def test_the_series_rounds_as_the_rates_it_projects(): @@ -3134,20 +3378,43 @@ def visit(node): rate=st.floats(min_value=0.5, max_value=2000, allow_nan=False), ) def test_the_importer_projects_rates_as_the_model_does(growth, offset, rate): - """Differential: compound_model_index equals the index - create_economic_assumption_indices builds from the same series, and - projected_rate_units, in pennies, equals the rate uprate_parameters - projects with the national living wage's metadata, in every year from - the one after the rate set to 2039.""" - set_until = min(growth) + offset - index = compound_model_index({date(year, 1, 1): v for year, v in growth.items()}) - assert index == model_index(growth) + """Differential, on generated series (up to 12 years from 2010 to 2030, + each growing -20% to +30% in millionths) and rates (50p to £2,000): + compound_model_index, compounded to where create_economic_assumption_indices + ends the index, equals the index that builder writes from the same series + in every year, and projected_rate_units, in pennies, equals the rate + uprate_parameters projects with the national living wage's metadata in + every year from the one after the rate set to the index's last.""" + model = model_index(growth) + set_until = min(min(growth) + offset, max(model)) + index = compound_model_index( + {date(year, 1, 1): v for year, v in growth.items()}, max(model) + ) + assert index == model wages = model_wages(growth, set_until, rate) - for year in range(set_until + 1, 2040): + assert set(wages) == set(range(set_until + 1, max(model) + 1)) + for year in wages: units = projected_rate_units(index, set_until, rate, year, 0.01) assert units * 0.01 == wages[year], year +def test_the_importer_index_is_the_models_for_the_committed_series(): + """Differential against the model itself: for the committed + yoy_growth.yaml, the index build_model_index reads from the model's own + generation steps runs to the year the model's builder ends it, and + compound_model_index compounds the same index from the committed + series. The processed parameters of the model hold that index on + 1 January of each of those years.""" + committed = get_yoy_growth_path().read_text() + model = build_model_index(committed, NLW_PATH) + written = _series(yaml.safe_load(committed), NLW_PATH)["values"] + assert compound_model_index(written, max(model)) == model + assert min(model) == min(written).year + processed = system.parameters.gov.economic_assumptions.indices.obr + for year, value in model.items(): + assert processed.national_living_wage(f"{year}-01-01") == value, year + + def edit_real_table_1_13(edits): """The real October 2026 workbook with cells of sheet 1.13 set: text as an inline string, a number as a number, None to empty the cell. It @@ -3230,15 +3497,16 @@ def edit_real_table_1_13(edits): ), "note_disagrees": ( {"B6": NLW_NOTE.format(set_until=2027, estimates_after=2028)}, - "does not say the figures beyond 2027 are OBR estimates", + "no statement in .* that the figures beyond 2027, and no others, are OBR " + "estimates", ), "set_beyond_the_table": ( {"B6": NLW_NOTE.format(set_until=2035, estimates_after=2035)}, - "up until 2035, outside its years 2020-2031", + "up until 2035, outside the table's years 2020-2031", ), "set_before_the_table": ( {"B6": NLW_NOTE.format(set_until=2019, estimates_after=2019)}, - "up until 2019, outside its years 2020-2031", + "up until 2019, outside the table's years 2020-2031", ), "note_sets_two_years": ( { @@ -3246,12 +3514,41 @@ def edit_real_table_1_13(edits): "beyond 2027 are OBR estimates. The NLW has been set by the " "Government up until 2028." }, - r"names more than one year the Government has set the rate up until, " + r"names more than one year the government has set the rate up until, " r"\[2027, 2028\]", ), "estimates_beyond_another_year": ( {"B7": "Figures beyond 2028 are OBR estimates."}, - r"and no others; it names \[2027, 2028\]", + r"the OBR names \[2027, 2028\]", + ), + "note_denies_the_year": ( + { + "B6": NLW_NOTE.format(set_until=2027, estimates_after=2027).replace( + "has been set", "has not been set" + ) + }, + "says the rate has not been set by the government up until a year", + ), + "note_gives_a_fiscal_year": ( + { + "B6": "The NLW has been set by the Government up until 2027\u201328. " + "Figures beyond 2027 are OBR estimates." + }, + "gives '2027\u201328', not one calendar year", + ), + "note_gives_a_spaced_range": ( + { + "B6": "The NLW has been set by the Government up until 2027 - 28. " + "Figures beyond 2027 are OBR estimates." + }, + "gives '2027 - 28', not one calendar year", + ), + "note_gives_a_five_digit_year": ( + { + "B6": "The NLW has been set by the Government up until 20270. " + "Figures beyond 2027 are OBR estimates." + }, + "gives '20270', not one calendar year", ), } @@ -3260,8 +3557,9 @@ def edit_real_table_1_13(edits): "edits, message", REAL_LAYOUT_CHANGES.values(), ids=REAL_LAYOUT_CHANGES.keys() ) def test_a_changed_layout_of_the_real_table_is_refused(edits, message): - """Each change to the real Table 1.13 that could put another row, unit - or year's figure into the series stops the importer instead.""" + """The importer refuses each change to the real Table 1.13 that could + put another row, unit or year's figure into the series, or leave the + OBR's note without one calendar year the government has set.""" with pytest.raises(ValueError, match=message): compute_level_growth(edit_real_table_1_13(edits)) @@ -3276,13 +3574,34 @@ def test_a_changed_layout_of_the_real_table_is_refused(edits, message): ids=["label_digit", "label_superscript", "title_and_unit_superscript"], ) def test_a_footnote_marker_is_read_past(edits): - """A footnote marker after the label's closing bracket, as a digit or a - superscript, or after the title or unit, leaves the table as it reads.""" + """The importer reads the same table when a digit or superscript + footnote marker follows the label's closing bracket, the title or the + unit.""" assert compute_level_growth(edit_real_table_1_13(edits)) == compute_level_growth( REAL_TABLE_1_13 ) +@pytest.mark.parametrize( + "note", + [ + NLW_NOTE.format(set_until=2027, estimates_after=2027), + "The NLW has been set by the Government up until 2027 inclusive. Figures " + "beyond 2027 are OBR estimates.", + "The NLW has been set by the Government up until 2027, following the " + "LPC's recommendations. Figures beyond 2027 are OBR estimates.", + "The NLW has been set by the Government up until 2027 Figures beyond " + "2027 are OBR estimates", + ], + ids=["october_2026", "inclusive", "comma", "no_full_stop"], +) +def test_an_affirmative_note_names_the_year_set(note): + """The importer reads 2027 from each wording that says the government + has set the rate up until 2027, the real October 2026 note among them.""" + line = compute_level_growth(edit_real_table_1_13({"B6": note}))[NLW_PATH].line + assert line.set_until == 2027 + + def test_the_real_table_reads_the_same_with_no_cell_changed(): assert compute_level_growth(edit_real_table_1_13({})) == compute_level_growth( REAL_TABLE_1_13 @@ -3313,13 +3632,13 @@ def test_the_labelled_row_is_read_not_the_row_below_the_years(): values: # From an older table 2021-01-01: 0.0100 - # Set by the Government (2024-2025) from OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)": £10.00 in 2023 to £11.00 in 2025 + # The government set the rates for 2024-2025: £10.00 in 2023 to £11.00 in 2025 (OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)") 2024-01-01: 0.0500 2025-01-01: 0.047619 - # OBR estimates (2026-2027) from OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)": £11.00 in 2025 to £12.00 in 2027 + # The OBR estimates the rates for 2026-2027: £11.00 in 2025 to £12.00 in 2027 (OBR EFO March 2026, economy Table 1.13 "National Living Wage (NLW)") 2026-01-01: 0.0500 2027-01-01: 0.038961 - # Beyond the OBR forecast: the average_earnings long-run path above, year for year + # After the OBR forecast, the importer copies average_earnings year for year 2028-01-01: 0.0000 2031-01-01: 0.0200 metadata: @@ -3333,16 +3652,25 @@ def test_the_labelled_row_is_read_not_the_row_below_the_years(): def render_nlw(content=YOY_GROWTH_FIXTURE, workbook=None, release=("March", 2026)): - growth = compute_level_growth(workbook or nlw_workbook()) - return render_level_series_yaml(content, growth, release) + """Render the national living wage from ``workbook`` (by default + nlw_rows' table) into a test file, from the rate its note implies + (table_anchor), with the index fixture_index builds.""" + workbook = workbook or nlw_workbook() + return render_level_series_yaml( + content, + compute_level_growth(workbook), + release, + {NLW_PATH: table_anchor(workbook)}, + fixture_index, + ) def test_render_writes_set_rates_estimates_then_average_earnings(): - """Years before the table's second year stay; the rates the Government - set and the OBR's estimates each get a comment naming their levels; then - average_earnings as the fixture holds it after 2027 (0% from 2026, 2% - from 2031), year for year. Only the first reference moves to the - release.""" + """The renderer keeps the years before the table's second year, labels + the rates the government set and the OBR's estimates with their end + levels, and then copies average_earnings as the fixture holds it after + 2027 (0% from 2026, 2% from 2031), year for year. It moves only the + first reference to the release.""" rendered = render_nlw() assert LEVEL_SERIES_FIXTURE in YOY_GROWTH_FIXTURE assert rendered == YOY_GROWTH_FIXTURE.replace(LEVEL_SERIES_FIXTURE, NLW_BLOCK) @@ -3396,7 +3724,7 @@ def test_main_writes_the_national_living_wage_from_the_economy_tables(tmp_path, 2028: 0.0333, 2031: 0.02, } - assert "set by the Government up until 2025" in capsys.readouterr().out + assert "the government has set rates up until 2025" in capsys.readouterr().out @pytest.mark.parametrize( @@ -3427,15 +3755,16 @@ def test_main_writes_nothing_without_a_readable_table(tmp_path, workbook, messag def test_the_summary_marks_a_growth_that_keeps_the_penny(capsys): """The summary main prints marks a year whose growth differs from its - levels' ratio: £13.525001 for 2028 after £13.21 set for 2027.""" + levels' ratio: £13.525001 for 2028 after the £13.21 the parameter file + sets for 2027.""" workbook = nlw_workbook(nlw_rows((13.21, 13.525001000000001), 2027, 2027)) growth = import_obr_forecasts.penny_preserving_level_growth( EARLY_EARNINGS_FIXTURE, compute_level_growth(workbook) ) import_obr_forecasts.print_level_summary(growth) assert ( - "2028: 0.023848 (not the levels' ratio, which projects another penny)" - in capsys.readouterr().out + "2028: 0.023848 (the importer moves it off the levels' ratio, which " + "projects another penny)" in capsys.readouterr().out ) @@ -3485,20 +3814,29 @@ def earnings_fixture(rate): first_year=2027, set_offset=0, ) +# Exactly £13.525 for 2028: policyengine-core rounds 1352.5 pennies to 1352. +@example( + first_level=13.21, + changes=[0.023845571536714447], + first_year=2027, + set_offset=0, +) def test_the_model_projects_the_published_levels_to_the_penny( first_level, changes, first_year, set_offset ): - """For any national living wage-like table (£5 to £20 in its first year, - up to 12 years, each changing by -5% to +10%): the importer reads each - year's growth as its level over the year before's, to six decimals; the - years up to the one the note names fall under the rates set and the rest - under the OBR's estimates; the index the model compounds from the - written series exceeds zero; and from the table's level in the year the - note names, the model projects every later level to the penny. The - importer writes another growth only for an estimate whose ratio, after - the years before as written, projects another penny. The note names a - year the series' index covers (test_a_table_needs_an_index_in_the_year_set - covers the others).""" + """For generated tables like the national living wage's (£5 to £20 in + the first year, from 2015 to 2027, then up to 11 more years, each + changing by -5% to +10%), with the rate set in a year the series' index + covers rounded to the penny as a parameter file holds it: the importer + reads each year's growth as its level over the year before's, to six + decimals; the years up to the one the note names fall under the rates + set and the rest under the OBR's estimates; the index the model + compounds from the written series exceeds zero; and from the rate set, + the model projects every later level to its penny, as policyengine-core + rounds it. The importer writes another growth only for an estimate whose + ratio, after the years before as written, projects another penny. + (test_a_table_needs_an_index_in_the_year_set covers a rate set before + the index starts.)""" levels = [first_level] for change in changes: levels.append(levels[-1] * (1 + change)) @@ -3519,29 +3857,30 @@ def test_the_model_projects_the_published_levels_to_the_penny( rendered = render_nlw(EARLY_EARNINGS_FIXTURE, workbook) values = _values(rendered, NLW_PATH) assert all(level > 0 for level in model_index(values).values()) - wages = model_wages(values, set_until, published[set_until]) + rate = table_anchor(workbook).rate + wages = model_wages(values, set_until, rate) + + def penny(level): + return round(level / 0.01) * 0.01 + for year in years: if year > set_until: - assert wages[year] == pytest.approx(round(published[year], 2), abs=1e-9) + assert wages[year] == pytest.approx(penny(published[year]), abs=1e-9) for year in years[1:]: if values[year] != ratios[year]: event("an estimate's growth changed") assert year > set_until - ratio_wages = model_wages( - {**values, year: ratios[year]}, set_until, published[set_until] - ) - assert ratio_wages[year] != pytest.approx( - round(published[year], 2), abs=1e-9 - ) + ratio_wages = model_wages({**values, year: ratios[year]}, set_until, rate) + assert ratio_wages[year] != pytest.approx(penny(published[year]), abs=1e-9) - block = rendered[rendered.index(" national_living_wage:") :] kind = None + block = rendered[rendered.index(" national_living_wage:") :] for line in block.split("\n")[: block.split("\n").index(" metadata:")]: - if line.strip().startswith("# Set by the Government"): + if line.strip().startswith("# The government set the rates for"): kind = "set" - elif line.strip().startswith("# OBR estimates"): + elif line.strip().startswith("# The OBR estimates the rates for"): kind = "estimate" - elif line.strip().startswith("# Beyond"): + elif line.strip().startswith("# After the OBR forecast"): kind = None elif (match := import_obr_forecasts.VALUE_LINE_RE.match(line)) and int( match.group(1) @@ -3551,8 +3890,9 @@ def test_the_model_projects_the_published_levels_to_the_penny( def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): - """The table keeps the index above zero; the average_earnings tail takes - it there, so the finished series' check stops the render.""" + """Growth through the table keeps the index above zero; the falling + average_earnings tail then takes it to zero, so the importer refuses the + render.""" workbook = nlw_workbook(nlw_rows((10.0, 10.0, 0.0001), 2023, 2023)) render_nlw(workbook=workbook) flat = FLAT_SERIES_FIXTURE.format(key="average_earnings") @@ -3565,11 +3905,11 @@ def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): def test_a_finished_index_that_overflows_is_refused(): - """£0.00001 for 2027 and £1.7e303 for 2028 give finite growth, 1.7e308, - which the table's own index check, from 1 in 2027, passes. In the - committed file the index exceeds 1 by 2027, so the finished series' - index overflows in 2028: the finished-index check refuses that series, - and the importer refuses to write the table.""" + """£0.00001 for 2027 and £1.7e303 for 2028 give finite growth, 1.7e308. + In the committed file the kept years take the index above 1 by 2027, so + that growth takes it to infinity in 2028: the importer refuses the + table, and check_index_stays_positive refuses the finished series it + would write.""" workbook = nlw_workbook(nlw_rows((1e-5, 1.7e303), 2027, 2027)) growth = compute_level_growth(workbook)[NLW_PATH] assert math.isfinite(growth.values[2028]) and growth.values[2028] > 1e308 @@ -3590,12 +3930,16 @@ def test_a_finished_index_that_overflows_is_refused(): ValueError, match="national_living_wage: the index reaches inf in 2028" ): import_obr_forecasts.check_index_stays_positive(NLW_PATH, finished) - with pytest.raises(ValueError, match="national_living_wage: no 2028 growth"): + with pytest.raises( + ValueError, + match=r"national_living_wage: 2028 growth of 1\.7e\+308 takes the index " + "to inf", + ): render_level_series_yaml(committed, {NLW_PATH: growth}, ("October", 2026)) def test_growth_past_every_float_is_refused(): - with pytest.raises(ValueError, match="takes the index to inf"): + with pytest.raises(ValueError, match="growth of inf, which the importer cannot"): compute_level_growth(nlw_workbook(nlw_rows((1e-300, 1e300), 2023, 2023))) @@ -3614,18 +3958,19 @@ def test_growth_past_every_float_is_refused(): levels=st.lists(TINY_TO_HUGE, min_size=2, max_size=8), earnings=st.sampled_from((0.02, -0.51)), ) -# A fall to a thousandth passes the table's check; the falling tail then -# takes the finished index to zero in 2037. +# A fall to a thousandth keeps the index above zero through the table; the +# falling tail then takes the finished index to zero in 2037. @example(levels=[1.0, 0.001], earnings=-0.51) def test_a_written_level_series_keeps_every_index_positive(levels, earnings): - """For any positive levels, however far apart: the importer refuses the - table exactly when its six-decimal growth takes the five-decimal index - the model compounds through it (from 1 in its first year) to zero, and - refuses to write the series exactly when the index of the finished - series, with the year kept before it and average_earnings (rising 2% or - falling 51% a year) after it, reaches zero by the end of the growth - file. Otherwise that index stays above zero in every year, and the - model's index agrees with it as far as the model compounds. The property + """For generated positive levels (2 to 8 years from 2023, drawn from + TINY_TO_HUGE) and an average_earnings tail rising 2% or falling 51% a + year: the importer refuses the render exactly when the five-decimal + index the model compounds through the written series (the years the + file keeps before the table, the table's six-decimal growth, then + average_earnings) reaches zero or infinity by LAST_INDEX_YEAR. It names + the table year where the index first does so, or the later year where + the finished series' index does. When it writes the series, the model's + index equals that index in every year the model builds. The property leaves the penny check out, as levels this far apart rarely round to a penny the index can reach.""" content = earnings_fixture(earnings) @@ -3634,40 +3979,45 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): 2023 + i: round(levels[i] / levels[i - 1] - 1, 6) + 0.0 for i in range(1, len(levels)) } - through_table = model_index({2023: 0.0, **values}) - try: - growth = compute_level_growth(workbook)[NLW_PATH] - except ValueError as error: - event("refused: table") - assert "which the model cannot compound from" in str(error) - assert any(through_table[year] <= 0 for year in values) - return - assert all(through_table[year] > 0 for year in values) - assert growth.values == values + assert compute_level_growth(workbook)[NLW_PATH].values == values with pytest.MonkeyPatch.context() as patch: patch.setattr( import_obr_forecasts, "penny_preserving_growth", - lambda _, series, __: series, + lambda _, series, *__: series, ) with pytest.MonkeyPatch.context() as unchecked_patch: - unchecked_patch.setattr( - import_obr_forecasts, "check_index_stays_positive", lambda *_: None - ) + for check in ( + "check_level_index", + "check_index_stays_positive", + "check_level_index_in_model", + ): + unchecked_patch.setattr(import_obr_forecasts, check, lambda *_: None) unchecked = render_nlw(content, workbook) written = _values(unchecked, NLW_PATH) finished = compounded(written, import_obr_forecasts.LAST_INDEX_YEAR) - # The model compounds the same index, as far as 2039. - model = model_index(written) - assert {year: finished[year] for year in model} == model - assert all(math.isfinite(level) for level in finished.values()) + failed = [ + year + for year, level in finished.items() + if not (math.isfinite(level) and level > 0) + ] try: rendered = render_nlw(content, workbook) except ValueError as error: - event("refused: finished series") - assert "national_living_wage: the index reaches" in str(error) - assert any(level <= 0 for level in finished.values()) + assert failed, error + if failed[0] in values: + event("refused: table") + assert f"{failed[0]} growth of {values[failed[0]]} takes the index" in ( + str(error) + ) + else: + event("refused: finished series") + assert f"the index reaches {finished[failed[0]]} in {failed[0]}" in str( + error + ) return event("written") + assert not failed assert rendered == unchecked - assert all(level > 0 for level in finished.values()) + model = model_index(written) + assert {year: finished[year] for year in model} == model diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index dae9dbe144..ab6d863b4e 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -591,9 +591,10 @@ def update_forecast_gap_yaml( TABLE_NUMBER_RE = re.compile(r"\d+\.\d+") FOOTNOTE_RE = re.compile(r"(?<=[a-z])\d+$", re.IGNORECASE) VALUE_LINE_RE = re.compile(r"^\s*(\d{4})-\d{2}-\d{2}:") -# The last year yoy_growth.yaml states. create_economic_assumption_indices -# stops at or before it, so a series that stays positive to here stays -# positive in the model. +# One year beyond 2073, the last year yoy_growth.yaml states. The model holds +# each series' last growth in later years, and create_economic_assumption_indices +# ends the committed file's indices by 2074, so a series whose index stays +# positive to here stays positive in the model. LAST_INDEX_YEAR = 2074 @@ -1695,8 +1696,8 @@ def render_receipts_growth_yaml( @dataclass(frozen=True) class LevelSeriesSpec: - """An economy table that publishes a series as levels across a row, not - as growth down a column. + """A series the OBR publishes as levels across a row of years, not as + growth down a column. ``path`` names the series under ``obr``. The importer finds the table by its title, as it finds receipts tables. The table's header declares one @@ -1704,18 +1705,23 @@ class LevelSeriesSpec: one row below it, labelled ``row``, holds a level under each year and nowhere else. The importer compares titles, units and labels lowercased, without a footnote marker: digits or superscript digits after a letter or - a closing bracket, as in "National Living Wage (NLW)1". A note that the - rate "has been set by the Government up until" a year, and that figures - beyond that year "are OBR estimates", splits the years the Government has - set from the OBR's estimates; every such statement in the table must name - the same year. The parameters the series projects round each projected - rate to the nearest ``rounding``. + a closing bracket, as in "National Living Wage (NLW)1". The OBR's note + says the rate "has been set by the Government up until" a year and that + "figures beyond" it "are OBR estimates"; the importer labels the years + on each side of that year, and every such statement in the table must + name the same calendar year. + + ``anchor`` names the scale whose last bracket's amount the series' index + uprates. That amount's last dated value is the rate the model projects + from, and its uprating rounds each projected rate to the nearest + ``rounding``. """ path: tuple[str, ...] title: str unit: str row: str + anchor: str rounding: float @@ -1725,22 +1731,28 @@ class LevelSeriesSpec: title="national living wage", unit="£ per hour", row="national living wage (nlw)", + # The national living wage is the scale's last bracket. + anchor="gov.hmrc.minimum_wage.non_apprentice", # gov.hmrc.minimum_wage rounds each projected rate to the nearest penny. rounding=0.01, ), ) # A footnote marker ending a level table's title, unit or label. -LEVEL_FOOTNOTE_RE = re.compile( - r"(?<=[a-z)\]])(?:\d+|[\u2070\u00b9\u00b2\u00b3\u2074-\u2079]+)$", re.IGNORECASE +LEVEL_FOOTNOTE_RE = re.compile(r"(?<=[a-z)\]])(?:\d+|[⁰¹²³⁴-⁹]+)$", re.IGNORECASE) +# A statement of the year the government has set the rate up until, with any +# negation before it and the whole token after it, and a range such as +# "2027 - 28" with its spaces. +SET_BY_GOVERNMENT_RE = re.compile( + r"(?:\b(not|never)\s+(?:yet\s+)?(?:been\s+)?)?" + r"set by the government up until (\S+(?:\s*[-–—]\s*\S+)?)" ) -SET_BY_GOVERNMENT_RE = re.compile(r"set by the government up until (\d{4})") OBR_ESTIMATES_RE = re.compile(r"figures beyond (\d{4}) are obr estimates") @dataclass(frozen=True) class LevelLine: - """A level series as one EFO publishes it: levels by calendar year, and - the last year whose level the Government has set.""" + """A level series as the OBR publishes it in one EFO: levels by calendar + year, and the last year whose rate the government has set.""" table: str label: str @@ -1752,22 +1764,35 @@ class LevelLine: class LevelGrowth: """Growth in a level series, by year, to six decimals, from the table's second year to its last. compute_level_growth writes each year's level - over the year before's; penny_preserving_growth then changes any OBR - estimate's growth that would project the wrong penny.""" + over the year before's; penny_preserving_growth then changes the growth + in any year after the anchor's that would project another penny.""" values: dict[int, float] line: LevelLine spec: LevelSeriesSpec +@dataclass(frozen=True) +class LevelAnchor: + """The rate a level series' index projects from: the last dated value of + the parameter it uprates, the date that value takes effect, and that + parameter's uprating metadata. policyengine-core's uprate_parameters + projects each later rate from this value.""" + + instant: date + rate: float + uprating: dict + + def level_label(value: str | None) -> str: - """A level table's title, unit or label as the importer compares it: - lowercased, with single spaces and without a footnote marker.""" + """Return a level table's title, unit or label as the importer compares + it: lowercased, with single spaces and without a footnote marker.""" return LEVEL_FOOTNOTE_RE.sub("", normalise_label(value)) def _year_columns(row: dict[str, str | None]) -> dict[str, int]: - """A row's columns and years, if every non-empty cell holds a year.""" + """Return a row's columns and years, if every non-empty cell holds a + year.""" cells = { column: str(value).strip() for column, value in row.items() @@ -1779,22 +1804,39 @@ def _year_columns(row: dict[str, str | None]) -> dict[str, int]: def _set_until(rows: list[dict[str, str | None]], where: str) -> int: - """The last year the Government has set, from the table's note. Every - statement in the table of the year the Government has set the rate up + """Read the last year the government has set from the OBR's note. Every + statement in the table of the year the government has set the rate up until, or of the year beyond which figures are OBR estimates, must name - the same year.""" + the same calendar year, and none may deny it.""" texts = [normalise_label(value) for row in rows for value in row.values()] notes = [text for text in texts if SET_BY_GOVERNMENT_RE.search(text)] if len(notes) != 1: raise ValueError( - f"{where} needs one note saying which years the Government has set " - f"and which are OBR estimates; found {len(notes)}" + f"The importer needs one note in {where} saying which years the " + f"government has set and which the OBR estimates; found {len(notes)}" ) - set_years = sorted({int(year) for year in SET_BY_GOVERNMENT_RE.findall(notes[0])}) + statements = [ + (negation, token.rstrip(".,;:)")) + for negation, token in SET_BY_GOVERNMENT_RE.findall(notes[0]) + ] + for negation, token in statements: + if negation: + raise ValueError( + f"The OBR's note in {where} says the rate has {negation} been " + f"set by the government up until a year, so it names no year " + f"the government has set: {notes[0]!r}" + ) + if not re.fullmatch(r"\d{4}", token): + raise ValueError( + f"The OBR's note in {where} gives {token!r}, not one calendar " + f"year, as the year the government has set the rate up until: " + f"{notes[0]!r}" + ) + set_years = sorted({int(token) for _, token in statements}) if len(set_years) != 1: raise ValueError( - f"{where}'s note names more than one year the Government has set " - f"the rate up until, {set_years}: {notes[0]!r}" + f"The OBR's note in {where} names more than one year the " + f"government has set the rate up until, {set_years}: {notes[0]!r}" ) set_until = set_years[0] estimates = sorted( @@ -1802,9 +1844,9 @@ def _set_until(rows: list[dict[str, str | None]], where: str) -> int: ) if estimates != [set_until]: raise ValueError( - f"{where} does not say the figures beyond {set_until} are OBR " - f"estimates, and no others; it names {estimates or 'no year'}: " - f"{notes[0]!r}" + f"The importer found no statement in {where} that the figures " + f"beyond {set_until}, and no others, are OBR estimates; the OBR " + f"names {estimates or 'no year'}: {notes[0]!r}" ) return set_until @@ -1812,8 +1854,8 @@ def _set_until(rows: list[dict[str, str | None]], where: str) -> int: def find_level_line( sheets: dict[str, list[dict[str, str | None]]], spec: LevelSeriesSpec ) -> LevelLine: - """The levels in the one row labelled ``spec.row`` of the one table - titled ``spec.title``, read across its row of years. + """Read the levels in the one row labelled ``spec.row`` of the one table + titled ``spec.title``, across its row of years. It refuses any other layout rather than read it: another unit, a second row of years or of the label, a level missing under a year or one with @@ -1893,8 +1935,9 @@ def find_level_line( set_until = _set_until(rows, where) if not min(levels) <= set_until <= max(levels): raise ValueError( - f"{where} says the Government has set the rate up until " - f"{set_until}, outside its years {min(levels)}-{max(levels)}" + f"The OBR's note in {where} says the government has set the rate up " + f"until {set_until}, outside the table's years " + f"{min(levels)}-{max(levels)}" ) if TABLE_NUMBER_RE.fullmatch(sheet): table = sheet @@ -1909,7 +1952,8 @@ def find_level_line( def compute_level_growth( xlsx_bytes: bytes, specs: tuple[LevelSeriesSpec, ...] = LEVEL_SERIES_SPECS ) -> dict[tuple[str, ...], LevelGrowth]: - """Growth in each level series from an EFO's economy tables.""" + """Read the growth in each level series from an EFO's economy tables: + each year's level over the year before's, less one, to six decimals.""" sheets = read_workbook_sheets(xlsx_bytes) result = {} for spec in specs: @@ -1919,89 +1963,139 @@ def compute_level_growth( year: round(line.levels[year] / line.levels[year - 1] - 1, 6) + 0.0 for year in sorted(line.levels)[1:] } - # As for the receipts series: the model compounds the growth into an - # index rounded to five decimals, which must stay above zero. - index = 1.0 for year, value in values.items(): - index = round(index * (1 + value), 5) - if not (math.isfinite(index) and index > 0): + if not math.isfinite(value): raise ValueError( - f"{name}: {year} growth of {value} takes the index to " - f"{index} at five decimals, which the model cannot " - "compound from" + f"{name}: the {year} level over the {year - 1} level gives " + f"growth of {value}, which the importer cannot write" ) result[spec.path] = LevelGrowth(values, line, spec) return result -# create_economic_assumption_indices compounds each index to 2039. -MODEL_INDEX_LAST_YEAR = 2039 +def read_level_anchor(spec: LevelSeriesSpec) -> LevelAnchor: + """Read the last dated value of ``spec.anchor``'s last bracket's amount, + and its uprating metadata, from its file in the model's parameter tree. + It refuses an amount that the series' index does not uprate to the + nearest ``spec.rounding``.""" + path = ( + get_repo_root() / "policyengine_uk/parameters" / Path(*spec.anchor.split(".")) + ) + amount = yaml.safe_load(path.with_suffix(".yaml").read_text())["brackets"][-1][ + "amount" + ] + uprating = amount["metadata"]["uprating"] + expected = { + "parameter": "gov.economic_assumptions.indices.obr." + ".".join(spec.path), + "rounding": {"type": "nearest", "interval": spec.rounding}, + } + if uprating != expected: + raise ValueError( + f"{spec.anchor}'s last bracket must uprate by {expected['parameter']} " + f"to the nearest {spec.rounding}; its metadata gives {uprating}" + ) + instant = max(amount["values"]) + value = amount["values"][instant] + return LevelAnchor( + instant=date.fromisoformat(str(instant)), + rate=float(value["value"] if isinstance(value, dict) else value), + uprating=uprating, + ) -def compound_model_index(values: dict) -> dict[int, float]: - """The index create_economic_assumption_indices compounds from a growth - series: 1 in its first listed year, then each year's index times one plus - the growth in force on 1 January, rounded to five decimals, to 2039.""" +def compound_model_index(values: dict, last_year: int) -> dict[int, float]: + """Compound a growth series' index as create_economic_assumption_indices + does, to ``last_year``: 1 in the series' first listed year, then each + year's index times one plus the growth in force on 1 January, rounded to + five decimals. The builder decides where an index ends (see + check_level_index_in_model).""" growth = {str(instant): value for instant, value in values.items()} first = int(min(growth)[:4]) index = {first: 1.0} - for year in range(first + 1, MODEL_INDEX_LAST_YEAR + 1): + for year in range(first + 1, last_year + 1): in_force = max(instant for instant in growth if instant <= f"{year}-01-01") index[year] = round(index[year - 1] * (1 + growth[in_force]), 5) return index def projected_rate_units( - index: dict[int, float], set_until: int, rate: float, year: int, rounding: float + index: dict[int, float], start: int, rate: float, year: int, rounding: float ) -> float: - """The rate uprate_parameters (policyengine-core) projects for ``year`` - from ``rate``, the last one set, in ``set_until``, as a whole number of - ``rounding``: the rate times the index's growth since, over ``rounding``, - rounded to the nearest whole number. Infinity if that overflows.""" - projected = rate * (index[year] / index[set_until]) / rounding + """Return the rate uprate_parameters (policyengine-core) projects for + ``year`` from ``rate``, in force in ``start``, as a whole number of + ``rounding``: the rate times the index's growth since, over + ``rounding``, rounded to the nearest whole number. Infinity if that + overflows.""" + projected = rate * (index[year] / index[start]) / rounding return round(projected) if math.isfinite(projected) else math.inf +def _level_growth_series(series: LevelGrowth, kept: dict) -> dict[str, float]: + """The growth the model reads for a level series: the years the file + keeps before the table, then the table's, keyed by date.""" + growth = {str(instant): value for instant, value in kept.items()} + growth.update({f"{year}-01-01": value for year, value in series.values.items()}) + return growth + + +def check_level_index(path: tuple[str, ...], series: LevelGrowth, kept: dict) -> None: + """Refuse a table whose growth takes the model's index to zero or + infinity. The model compounds ``kept``, the years the file keeps before + the table, and then the table's growth (compound_model_index), so the + importer compounds the same series.""" + index = compound_model_index(_level_growth_series(series, kept), max(series.values)) + for year, value in sorted(series.values.items()): + if not (math.isfinite(index[year]) and index[year] > 0): + raise ValueError( + f"{'.'.join(path)}: {year} growth of {value} takes the index to " + f"{index[year]} at five decimals, which the model cannot " + "compound from" + ) + + def penny_preserving_growth( - path: tuple[str, ...], series: LevelGrowth, kept: dict + path: tuple[str, ...], series: LevelGrowth, kept: dict, anchor: LevelAnchor ) -> LevelGrowth: - """``series`` with each OBR estimate's growth chosen so that the model - projects the table's level, as both round to the nearest + """Choose the growth in each table year after ``anchor``'s so that the + model projects the table's level, both rounded to the nearest ``series.spec.rounding``. The model compounds ``kept``, the earlier years the file keeps, and then - ``series`` into an index (compound_model_index), and projects each rate - after the last one set from it (projected_rate_units). The importer takes - the table's level in the year the Government has set the rate up until - as that rate. Each later year keeps its growth if the projection rounds - to the table's level. Otherwise the importer takes the growth, to six - decimals, from the level the index carries the rate to in the year before - to the table's level, or else the nearest six-decimal growth to that - which does round to the table's level. It refuses a year that no - six-decimal growth can reproduce. + ``series`` into an index (compound_model_index), and projects each later + rate from the anchor's rate (projected_rate_units). The importer refuses + a table whose level in the anchor's year differs from the anchor's rate + at that rounding, and one whose index starts after the anchor's year. + Each later year keeps its growth if the projection rounds to the table's + level. Otherwise the importer takes the growth, to six decimals, from + the level the index carries the rate to in the year before to the + table's level, or else the nearest six-decimal growth to that which does + round to the table's level. It refuses a year that no six-decimal growth + can reproduce. """ name = ".".join(path) line = series.line rounding = series.spec.rounding - set_until, last = line.set_until, max(line.levels) - if last > MODEL_INDEX_LAST_YEAR: + start, rate, last = anchor.instant.year, anchor.rate, max(line.levels) + if start in line.levels and round(line.levels[start] / rounding) != round( + rate / rounding + ): raise ValueError( - f"{name}: the model compounds its indices to {MODEL_INDEX_LAST_YEAR}, " - f"so it cannot project the {last} level" + f"{name}: the OBR's {start} level, {line.levels[start]!r}, differs " + f"to the nearest {rounding} from {rate!r}, the rate " + f"{series.spec.anchor} holds from {anchor.instant}" ) values = dict(series.values) - growth = {str(instant): value for instant, value in kept.items()} - growth.update({f"{year}-01-01": value for year, value in values.items()}) + growth = _level_growth_series(series, kept) first = int(min(growth)[:4]) - if set_until < first: + if start < first: raise ValueError( - f"{name}: the index starts in {first}, after {set_until}, the last " - "year the Government has set, so the model cannot project from it" + f"{name}: the index starts in {first}, after {start}, the year of " + f"the last rate {series.spec.anchor} holds, so the model cannot " + "project from it" ) - rate = line.levels[set_until] - for year in range(set_until + 1, last + 1): - index = compound_model_index(growth) - for base in (set_until, year - 1): + for year in [year for year in sorted(line.levels) if year > start]: + index = compound_model_index(growth, last) + for base in (start, year - 1): if not (math.isfinite(index[base]) and index[base] > 0): raise ValueError( f"{name}: the index reaches {index[base]} in {base}, so the " @@ -2009,8 +2103,8 @@ def penny_preserving_growth( ) refusal = ValueError( f"{name}: no {year} growth to six decimals lets the model project " - f"the table's {year} level, {line.levels[year]!r}, from its " - f"{set_until} rate to the nearest {rounding}" + f"the OBR's {year} level, {line.levels[year]!r}, from the {start} " + f"rate to the nearest {rounding}" ) if not math.isfinite(line.levels[year] / rounding): raise refusal @@ -2021,8 +2115,8 @@ def units(micros: int) -> float: value = micros / 1_000_000 except OverflowError: return math.inf - trial = compound_model_index({**growth, f"{year}-01-01": value}) - return projected_rate_units(trial, set_until, rate, year, rounding) + trial = compound_model_index({**growth, f"{year}-01-01": value}, last) + return projected_rate_units(trial, start, rate, year, rounding) written = round(Decimal(repr(values[year])) * 1_000_000) if units(written) == target: @@ -2048,7 +2142,7 @@ def first_above(bound: float) -> int: lowest, highest = first_above(target - 1), first_above(target) - 1 if lowest > highest: raise refusal - carried = rate * (index[year - 1] / index[set_until]) + carried = rate * (index[year - 1] / index[start]) ratio = line.levels[year] / carried - 1 sequential = round(ratio * 1e6) if math.isfinite(ratio * 1e6) else lowest values[year] = min(max(sequential, lowest), highest) / 1_000_000 + 0.0 @@ -2069,23 +2163,118 @@ def _kept_level_values(obr: dict, path: tuple[str, ...], series: LevelGrowth) -> } +def _level_anchor( + path: tuple[str, ...], + series: LevelGrowth, + anchors: dict[tuple[str, ...], LevelAnchor] | None, +) -> LevelAnchor: + """``anchors[path]`` if given, or else the anchor read_level_anchor + reads from the series' parameter file.""" + return anchors[path] if anchors is not None else read_level_anchor(series.spec) + + def penny_preserving_level_growth( - content: str, growth: dict[tuple[str, ...], LevelGrowth] + content: str, + growth: dict[tuple[str, ...], LevelGrowth], + anchors: dict[tuple[str, ...], LevelAnchor] | None = None, ) -> dict[tuple[str, ...], LevelGrowth]: - """penny_preserving_growth for each level series, given the years before - its table that ``content``, a yoy_growth.yaml, keeps.""" + """Run penny_preserving_growth on each level series, given the years + before its table that ``content``, a yoy_growth.yaml, keeps, and its + anchor (``anchors``, or by default read_level_anchor).""" obr = yaml.safe_load(content)["obr"] return { path: penny_preserving_growth( - path, series, _kept_level_values(obr, path, series) + path, + series, + _kept_level_values(obr, path, series), + _level_anchor(path, series, anchors), ) for path, series in growth.items() } +def build_model_index(content: str, path: tuple[str, ...]) -> dict[int, float]: + """Build, by year, the index the model compounds from ``obr.`` when + ``content`` is its yoy_growth.yaml. + + process_parameter_tree runs its generation steps on the parameters the + YAML files hold. This runs those steps, up to and including + create_economic_assumption_indices, on the same parameters with + ``content``'s growth series in place of the file's, and reads the index + that builder writes. The steps before it add growth series of their own, + and the builder decides where each index ends. + """ + from policyengine_core.parameters import ParameterNode, config + + from policyengine_uk.parameters.gov.economic_assumptions.create_economic_assumption_indices import ( + create_economic_assumption_indices, + ) + from policyengine_uk.tax_benefit_system import ( + GENERATION_STEPS, + statutory_parameters, + ) + + parameters = statutory_parameters() + assumptions = parameters.gov.economic_assumptions + del assumptions.children["yoy_growth"] + assumptions.add_child( + "yoy_growth", + ParameterNode( + "gov.economic_assumptions.yoy_growth", + data=yaml.load(content, Loader=config.Loader), + ), + ) + for step in GENERATION_STEPS: + parameters = step(parameters) + if step is create_economic_assumption_indices: + break + index = parameters.get_child( + "gov.economic_assumptions.indices.obr." + ".".join(path) + ) + return { + int(entry.instant_str[:4]): entry.value + for entry in sorted(index.values_list, key=lambda entry: entry.instant_str) + } + + +def check_level_index_in_model( + content: str, + path: tuple[str, ...], + series: LevelGrowth, + build_index=None, +) -> None: + """Check the index the model builds from ``content``'s ``obr.`` + against the importer's. ``build_index(content, path)`` builds it; by + default build_model_index does, as the model does. The index must reach + the table's last year, and compound_model_index must compound the same + index from the written series in every year the model's has.""" + name = ".".join(path) + model = (build_index or build_model_index)(content, path) + last = max(series.line.levels) + if max(model) < last: + raise ValueError( + f"{name}: the model's index ends in {max(model)}, so it cannot " + f"project the {last} level" + ) + node = yaml.safe_load(content)["obr"] + for key in path: + node = node[key] + importer = compound_model_index(node["values"], max(model)) + differ = sorted( + year + for year in model.keys() | importer.keys() + if model.get(year) != importer.get(year) + ) + if differ: + raise ValueError( + f"{name}: compound_model_index and the model's index differ in " + f"{differ}, so the importer cannot check the model's pennies" + ) + + def average_earnings_years(content: str, after_year: int) -> dict[int, float]: - """``average_earnings`` after ``after_year``, year for year: the path - ``average_earnings_path`` gives, with an entry in every year + """Return ``average_earnings`` after ``after_year``, year for year: the + path ``average_earnings_path`` gives, with an entry in every year average_earnings has one, so the two series list the same years.""" values = yaml.safe_load(content)["obr"]["average_earnings"]["values"] later = { @@ -2103,9 +2292,9 @@ def render_level_values( tail: dict[int, float], release: tuple[str, int], ) -> list[str]: - """A level series' values entries: earlier years as they were, then - the growth into the rates the Government has set, then into the OBR's - estimates, then the average_earnings path.""" + """Render a level series' values entries: earlier years as they were, + then the growth into the rates the government has set, then into the + OBR's estimates, then the average_earnings path.""" first_year = min(growth.values) kept: list[str] = [] pending: list[str] = [] @@ -2129,24 +2318,26 @@ def render_level_values( ) for kind, years in ( ( - "Set by the Government", + "The government set the rates for", [y for y in growth.values if y <= published.set_until], ), - ("OBR estimates", [y for y in growth.values if y > published.set_until]), + ( + "The OBR estimates the rates for", + [y for y in growth.values if y > published.set_until], + ), ): if not years: continue span = f"{years[0]}" if len(years) == 1 else f"{years[0]}-{years[-1]}" start, end = years[0] - 1, years[-1] lines.append( - f"{pad}# {kind} ({span}) from {source}: " - f"£{published.levels[start]:.2f} in {start} to " - f"£{published.levels[end]:.2f} in {end}" + f"{pad}# {kind} {span}: £{published.levels[start]:.2f} in {start} " + f"to £{published.levels[end]:.2f} in {end} ({source})" ) lines += [f"{pad}{y}-01-01: {format_growth(growth.values[y])}" for y in years] lines.append( - f"{pad}# Beyond the OBR forecast: the average_earnings long-run path " - "above, year for year" + f"{pad}# After the OBR forecast, the importer copies average_earnings " + "year for year" ) lines += [f"{pad}{y}-01-01: {format_growth(v)}" for y, v in tail.items()] return lines @@ -2156,27 +2347,38 @@ def render_level_series_yaml( content: str, growth: dict[tuple[str, ...], LevelGrowth], release: tuple[str, int], + anchors: dict[tuple[str, ...], LevelAnchor] | None = None, + build_index=None, ) -> str: """Rewrite each level series' values and first reference in a yoy_growth.yaml. It keeps the years before the table's second year, so the series' index - keeps its base year, and the series' other references. It writes the - growth penny_preserving_growth chooses, so the model projects each of the - OBR's estimates to the table's penny. Years after the table follow - average_earnings as ``content`` holds it. It checks that the finished - series' index stays above zero, and that the result differs from - ``content`` only in those values and that reference. + keeps its base year, and the series' other references. It refuses a + table whose growth takes the index to zero (check_level_index), and + writes the growth penny_preserving_growth chooses from each series' + anchor (``anchors``, or by default read_level_anchor), so the model + projects each table level after the anchor's year to the table's penny. + Years after the table follow average_earnings as ``content`` holds it. + It checks that the finished series' index stays above zero, that the + result differs from ``content`` only in those values and that reference, + and that the index the model builds from the result + (check_level_index_in_model, with ``build_index``) matches the + importer's. """ month, year = release expected = copy.deepcopy(yaml.safe_load(content)) lines = content.split("\n") + written = [] for path, series in growth.items(): node = expected["obr"] for key in path: node = node[key] kept = _kept_level_values(expected["obr"], path, series) - series = penny_preserving_growth(path, series, kept) + check_level_index(path, series, kept) + series = penny_preserving_growth( + path, series, kept, _level_anchor(path, series, anchors) + ) tail = average_earnings_years(content, max(series.values)) start, end = _series_block(lines, ("obr", *path)) indent = 2 * len(path) + 2 @@ -2223,6 +2425,7 @@ def render_level_series_yaml( } check_index_stays_positive(path, node["values"]) node["metadata"]["reference"][0] = reference + written.append((path, series)) rendered = "\n".join(lines) if yaml.safe_load(rendered) != expected: @@ -2230,6 +2433,8 @@ def render_level_series_yaml( "Rewriting the level series changed more than their values and " "first reference" ) + for path, series in written: + check_level_index_in_model(rendered, path, series, build_index) return rendered @@ -2453,7 +2658,7 @@ def print_receipts_summary( def print_level_summary(growth: dict[tuple[str, ...], LevelGrowth]) -> None: - print(f"Updating {len(growth)} OBR level series") + print(f"The importer reads {len(growth)} OBR level series") for path, series in growth.items(): levels = series.line.levels window = ", ".join( @@ -2461,13 +2666,14 @@ def print_level_summary(growth: dict[tuple[str, ...], LevelGrowth]) -> None: + ( "" if value == round(levels[year] / levels[year - 1] - 1, 6) + 0.0 - else " (not the levels' ratio, which projects another penny)" + else " (the importer moves it off the levels' ratio, which " + "projects another penny)" ) for year, value in sorted(series.values.items()) ) print( - f"- {'.'.join(path)} (Table {series.line.table}, set by the " - f"Government up until {series.line.set_until}): {window}" + f"- {'.'.join(path)} (Table {series.line.table}; the government " + f"has set rates up until {series.line.set_until}): {window}" ) @@ -2736,8 +2942,9 @@ def main(argv: list[str] | None = None) -> int: print_receipts_summary( receipts_growth, population_growth(source_yaml_path.read_text()) ) - # The economy tables hold the national living wage's levels, so it moves - # with the economy series unless skipped. + # The OBR publishes the national living wage's levels in the economy + # tables, so the importer refreshes it with the economy series unless + # --skip-national-living-wage leaves it alone. level_growth = None if not args.gaps_only and not args.skip_national_living_wage: level_growth = compute_level_growth(workbook_bytes) @@ -2782,8 +2989,9 @@ def main(argv: list[str] | None = None) -> int: ) if level_growth is not None: yoy_path = args.output or source_yaml_path - # The growth that keeps each penny depends on the years the file keeps - # before the table, so the summary waits for the file. + # The importer chooses the growth that keeps each penny from the years + # the rendered file keeps before the table, so it prints the summary + # after rendering the other series. print_level_summary( penny_preserving_level_growth(outputs[yoy_path], level_growth) ) From e43fd2e201210ec47329d6eeae494b047ff66191 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 18:06:05 -0400 Subject: [PATCH 13/22] Name who acts in the national living wage docs and messages The README says the government sets each rate, the OBR publishes the rates set and its estimates, and the model projects from the last rate in non_apprentice.yaml; it lists every refusal. Docstrings open with their action, and the layout messages name the importer rather than giving the table a verb. The changelog fragment states what the importer does now. Co-Authored-By: Claude Opus 5.5 --- changelog.d/2721.added.md | 2 +- .../gov/economic_assumptions/README.md | 63 ++++++++++++------- .../tests/test_import_obr_forecasts.py | 20 +++--- .../tests/test_minimum_wage_projection.py | 5 +- policyengine_uk/utils/import_obr_forecasts.py | 40 ++++++------ 5 files changed, 73 insertions(+), 57 deletions(-) diff --git a/changelog.d/2721.added.md b/changelog.d/2721.added.md index 3644cac72e..e063793f0b 100644 --- a/changelog.d/2721.added.md +++ b/changelog.d/2721.added.md @@ -1 +1 @@ -Taught the OBR forecasts importer to write the national living wage series from the EFO economy tables' national living wage table (Table 1.13). It reads the levels across the table's row of years, refuses any other layout, and refreshes the minimum wage projection with each EFO. +The OBR forecasts importer now writes national living wage growth from the EFO economy tables' national living wage table (Table 1.13). It reads the OBR's levels across the table's row of years, projects each estimate from the last national living wage rate in the minimum wage parameters as the model does, and writes growth that keeps each estimate's penny. diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index 9017cca8dd..0c8589ddee 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -67,29 +67,46 @@ for each duty's rates, the higher-stake share Machine Games Duty's rate assumes, and the approximations. `national_living_wage` projects the minimum wage rates -(`gov.hmrc.minimum_wage`) after the last rate set. It holds the growth of the -rates the Government has set, then of the OBR's estimates, both from the -national living wage table of the detailed economy tables (Table 1.13 in -October 2026), then `average_earnings`' long-run growth, year for year. HM -Treasury costed Budget 2026's Carer's Allowance taper on the OBR's estimates. +(`gov.hmrc.minimum_wage`) after the last rate set. The government sets each +rate following the Low Pay Commission's recommendations, and the OBR +publishes the rates the government has set, with its estimates of later +ones, in the national living wage table of the detailed economy tables +(Table 1.13 in October 2026). The series holds the growth into each of those +rates, then `average_earnings`' long-run growth, year for year. HM Treasury +costed Budget 2026's Carer's Allowance taper on the OBR's estimates. + `utils/import_obr_forecasts.py` writes the series with the other economy series. It finds the table by its title, reads the level under each year across the table's one row of years, and writes each year's level over the -year before's, to six decimals. The table's note says up to which year the -Government has set the rate, and the importer writes those years and the -OBR's estimates under separate comments. It then projects each estimate as -the model does, from the last rate set through the index to the nearest -penny. Where a year's ratio would leave the model a penny off the table, it -writes the six-decimal growth that keeps the table's penny. The importer stops -before writing any file if the table declares a unit other than £ per hour, -or lacks one row of years and one row labelled "National Living Wage (NLW)". -It also stops if the years skip or repeat, if a year lacks a positive level, -if a figure has no year above it, if the table lacks that note or its notes -name different years, or if no six-decimal growth keeps an estimate's penny. -`test_minimum_wage_projection.py` -holds the October 2026 table, checks it against the importer's fixture, and -fails if the series stops reproducing the table to the penny, or if its -long-run years stop matching `average_earnings`. +year before's, to six decimals. The OBR identifies in its note the last year +the government has set, and the importer labels the growth into those years +and into the OBR's estimates with separate comments. The model projects the +national living wage, the last bracket of +`gov/hmrc/minimum_wage/non_apprentice.yaml`, from that bracket's last rate, +through the index it builds from this series, to the nearest penny. The +importer projects each later year of the table from the same rate in the same +way, and where a year's ratio would leave the model a penny off the OBR's +level, it writes the six-decimal growth that keeps the OBR's penny. It then +runs the model's own generation steps on the file it would write, and checks +that the model's index reaches the table's last year and matches the index +the importer compounded. + +The importer stops before writing any file when: + +- the unit differs from £ per hour, or the table lacks one row of years or + one row labelled "National Living Wage (NLW)"; +- the years skip or repeat, a year lacks a positive level, or a figure has no + year above it; +- the OBR's note names no single calendar year the government has set, or its + statements name different years; +- the OBR's level in the year of the file's last rate differs from that rate + at the penny; +- the table's growth takes the model's index to zero or infinity; or +- no six-decimal growth keeps an estimate's penny. + +`test_minimum_wage_projection.py` holds the October 2026 table, checks it +against the importer's fixture, and fails if the series stops reproducing the +table to the penny, or if its long-run years stop matching `average_earnings`. ## Time horizons @@ -502,9 +519,9 @@ the forecast levels, though not their growth. EFO's policy tables for any new measure in the betting and gaming lines. Then update the published levels in `policyengine_uk/tests/test_gambling_duty_base_projection.py`. -7. The same run refreshes `national_living_wage` from the new economy - tables' national living wage table (Table 1.13 in October 2026), with - `average_earnings` after it. Then replace the cut-down table in +7. The importer refreshes `national_living_wage` in the same run, from the + new economy tables' national living wage table (Table 1.13 in October + 2026), with `average_earnings` after it. Then replace the cut-down table in `policyengine_uk/tests/fixtures/obr_economy/` (see its README) and the levels in `OBR_NATIONAL_LIVING_WAGE` in `policyengine_uk/tests/test_minimum_wage_projection.py`. Pass diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index b5c1357443..6419ac6914 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -2977,8 +2977,8 @@ def model_wages(values, set_until, rate): def wages_in_the_model(content): """The national living wage the model projects in April of each year - after the rate the parameter file sets last, when ``content`` is its - yoy_growth.yaml: the index the model's own generation steps build + after the rate the parameter file sets last, when the model reads + ``content`` as its yoy_growth.yaml: the index the model's own generation steps build (build_model_index), carried from the file's rate by policyengine-core's uprate_parameters.""" return uprated_wages( @@ -3034,7 +3034,7 @@ def test_levels_laid_out_down_a_column_are_refused(): **{4 + i: {"B": 2023 + i, "C": level} for i, level in enumerate(NLW_LEVELS)}, 10: {"B": NLW_NOTE.format(set_until=2025, estimates_after=2025)}, } - with pytest.raises(ValueError, match="one row of years across its columns"): + with pytest.raises(ValueError, match="one row of years across the columns of"): compute_level_growth(nlw_workbook(rows)) @@ -3418,7 +3418,7 @@ def test_the_importer_index_is_the_models_for_the_committed_series(): def edit_real_table_1_13(edits): """The real October 2026 workbook with cells of sheet 1.13 set: text as an inline string, a number as a number, None to empty the cell. It - copies every other archive entry as it was.""" + copies every other archive entry unchanged.""" output = BytesIO() with ZipFile(BytesIO(REAL_TABLE_1_13)) as source, ZipFile(output, "w") as dest: path = _sheet_paths(source)["1.13"] @@ -3458,10 +3458,10 @@ def edit_real_table_1_13(edits): ), "unit_in_pence": ( {"C3": "pence per hour"}, - "must declare its unit as '£ per hour'", + "needs the unit '£ per hour', and no other", ), - "second_unit": ({"D3": "per cent"}, "must declare its unit"), - "no_unit": ({"C3": None}, "must declare its unit"), + "second_unit": ({"D3": "per cent"}, "needs the unit"), + "no_unit": ({"C3": None}, "needs the unit"), "row_relabelled": ( {"B5": "National Living Wage"}, r'Expected one "national living wage \(nlw\)" row .* found 0', @@ -3484,11 +3484,11 @@ def edit_real_table_1_13(edits): "years_backwards": ({"C4": 2031, "N4": 2020}, "years skip, repeat or run"), "second_row_of_years": ( {"C8": 2020, "D8": 2021}, - "one row of years across its columns; found 2", + "one row of years across the columns of .*; found 2", ), "labelled_row_of_years": ( {"B4": "Calendar year"}, - "one row of years across its columns; found 0", + "one row of years across the columns of .*; found 0", ), "no_note": ({"B6": None}, "needs one note .* found 0"), "second_note": ( @@ -3740,7 +3740,7 @@ def test_main_writes_the_national_living_wage_from_the_economy_tables(tmp_path, drop={"1.13"}, first={"1.13": nlw_sheet({**nlw_rows(), 3: {"C": "pence per hour"}})}, ), - "must declare its unit as '£ per hour'", + "needs the unit '£ per hour', and no other", ), ], ids=["no_table", "unit_in_pence"], diff --git a/policyengine_uk/tests/test_minimum_wage_projection.py b/policyengine_uk/tests/test_minimum_wage_projection.py index 675e779d29..5d7766fe74 100644 --- a/policyengine_uk/tests/test_minimum_wage_projection.py +++ b/policyengine_uk/tests/test_minimum_wage_projection.py @@ -157,9 +157,8 @@ def test_the_series_reproduces_the_obr_table(): def test_the_table_here_is_the_one_the_importer_reads(): - """OBR_NATIONAL_LIVING_WAGE is Table 1.13 as the importer reads it from - the real October 2026 workbook, so a refreshed fixture fails here until - the table here moves with it.""" + """OBR_NATIONAL_LIVING_WAGE holds the levels the importer reads from the + real October 2026 fixture. Refresh both copies together.""" line = compute_level_growth(TABLE_1_13.read_bytes())[("national_living_wage",)].line assert line.levels == OBR_NATIONAL_LIVING_WAGE assert line.set_until == 2027 diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 3fd0438d72..4d318c094e 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1700,9 +1700,9 @@ class LevelSeriesSpec: growth down a column. ``path`` names the series under ``obr``. The importer finds the table by - its title, as it finds receipts tables. The table's header declares one - unit, ``unit``; one row of cells holds consecutive calendar years; and - one row below it, labelled ``row``, holds a level under each year and + its title, as it finds receipts tables. The OBR gives one unit, ``unit``, + in the table's header; one row of cells holds consecutive calendar years; + and one row below it, labelled ``row``, holds a level under each year and nowhere else. The importer compares titles, units and labels lowercased, without a footnote marker: digits or superscript digits after a letter or a closing bracket, as in "National Living Wage (NLW)1". The OBR's note @@ -1712,9 +1712,8 @@ class LevelSeriesSpec: name the same calendar year. ``anchor`` names the scale whose last bracket's amount the series' index - uprates. That amount's last dated value is the rate the model projects - from, and its uprating rounds each projected rate to the nearest - ``rounding``. + uprates. The model projects that amount from its last dated value, and + its uprating rounds each projected rate to the nearest ``rounding``. """ path: tuple[str, ...] @@ -1731,7 +1730,7 @@ class LevelSeriesSpec: title="national living wage", unit="£ per hour", row="national living wage (nlw)", - # The national living wage is the scale's last bracket. + # The scale's last bracket holds the national living wage. anchor="gov.hmrc.minimum_wage.non_apprentice", # gov.hmrc.minimum_wage rounds each projected rate to the nearest penny. rounding=0.01, @@ -1774,8 +1773,8 @@ class LevelGrowth: @dataclass(frozen=True) class LevelAnchor: - """The rate a level series' index projects from: the last dated value of - the parameter it uprates, the date that value takes effect, and that + """The rate the model projects a level series' parameter from: that + parameter's last dated value, the date it takes effect, and the parameter's uprating metadata. policyengine-core's uprate_parameters projects each later rate from this value.""" @@ -1876,7 +1875,8 @@ def find_level_line( year_rows = [index for index, row in enumerate(rows) if _year_columns(row)] if len(year_rows) != 1: raise ValueError( - f"{where} needs one row of years across its columns; found {len(year_rows)}" + "The importer needs one row of years across the columns of " + f"{where}; found {len(year_rows)}" ) header = year_rows[0] years = _year_columns(rows[header]) @@ -1907,8 +1907,8 @@ def find_level_line( ] if units != [spec.unit]: raise ValueError( - f"{where} must declare its unit as {spec.unit!r} above its years; " - f"found {units}" + f"The importer needs the unit {spec.unit!r}, and no other, above the " + f"years in {where}; found {units}" ) ordered = [years[column] for column in sorted(years, key=column_index)] if ordered != list(range(ordered[0], ordered[0] + len(ordered))): @@ -2031,8 +2031,8 @@ def projected_rate_units( def _level_growth_series(series: LevelGrowth, kept: dict) -> dict[str, float]: - """The growth the model reads for a level series: the years the file - keeps before the table, then the table's, keyed by date.""" + """Return the growth the model reads for a level series, keyed by date: + the years the file keeps before the table, then the table's.""" growth = {str(instant): value for instant, value in kept.items()} growth.update({f"{year}-01-01": value for year, value in series.values.items()}) return growth @@ -2151,8 +2151,8 @@ def first_above(bound: float) -> int: def _kept_level_values(obr: dict, path: tuple[str, ...], series: LevelGrowth) -> dict: - """The values a yoy_growth.yaml's ``obr`` mapping holds for a level - series before its table's second year, which a render keeps.""" + """Return the values a yoy_growth.yaml's ``obr`` mapping holds for a + level series before its table's second year, which a render keeps.""" node = obr for key in path: node = node[key] @@ -2168,8 +2168,8 @@ def _level_anchor( series: LevelGrowth, anchors: dict[tuple[str, ...], LevelAnchor] | None, ) -> LevelAnchor: - """``anchors[path]`` if given, or else the anchor read_level_anchor - reads from the series' parameter file.""" + """Return ``anchors[path]`` if given, or else the anchor + read_level_anchor reads from the series' parameter file.""" return anchors[path] if anchors is not None else read_level_anchor(series.spec) @@ -2195,7 +2195,7 @@ def penny_preserving_level_growth( def build_model_index(content: str, path: tuple[str, ...]) -> dict[int, float]: """Build, by year, the index the model compounds from ``obr.`` when - ``content`` is its yoy_growth.yaml. + it reads ``content`` as its yoy_growth.yaml. process_parameter_tree runs its generation steps on the parameters the YAML files hold. This runs those steps, up to and including @@ -2292,7 +2292,7 @@ def render_level_values( tail: dict[int, float], release: tuple[str, int], ) -> list[str]: - """Render a level series' values entries: earlier years as they were, + """Render a level series' values entries: the earlier years it keeps, then the growth into the rates the government has set, then into the OBR's estimates, then the average_earnings path.""" first_year = min(growth.values) From 5d2c2daeda3012db481a013d3be27a9b111f3b59 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 18:11:35 -0400 Subject: [PATCH 14/22] Test that the builder, not the importer, ends the index On this head create_economic_assumption_indices always ends the index in 2039, so only builders that end it elsewhere separate a hard-coded year from the builder's: one ending in 2026 must refuse a table running to 2027, and one running to 2045 must leave the render unchanged. The footnote and range regexes go back to writing their characters as escapes. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 38 +++++++++++++++++++ policyengine_uk/utils/import_obr_forecasts.py | 6 ++- 2 files changed, 42 insertions(+), 2 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 6419ac6914..6900f3c594 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3398,6 +3398,44 @@ def test_the_importer_projects_rates_as_the_model_does(growth, offset, rate): assert units * 0.01 == wages[year], year +def test_the_builder_decides_where_the_index_ends(): + """The importer takes the index's last year from the builder. With a + builder that ends the index in 2026, the model has no index to project + the default table's 2027 estimate, so the importer refuses the table; + with one that compounds the same series to 2045, compound_model_index + compounds to 2045 too and the importer writes it.""" + workbook = nlw_workbook() + anchors = {NLW_PATH: table_anchor(workbook)} + render = partial( + render_level_series_yaml, + YOY_GROWTH_FIXTURE, + compute_level_growth(workbook), + ("March", 2026), + anchors, + ) + + def ending_in_2026(content, path): + return { + year: value + for year, value in fixture_index(content, path).items() + if year <= 2026 + } + + with pytest.raises( + ValueError, + match="the model's index ends in 2026, so it cannot project the 2027 level", + ): + render(build_index=ending_in_2026) + + def running_to_2045(content, path): + index = fixture_index(content, path) + values = _series(yaml.safe_load(content), path)["values"] + assert compound_model_index(values, max(index)) == index + return compound_model_index(values, 2045) + + assert render(build_index=running_to_2045) == render_nlw() + + def test_the_importer_index_is_the_models_for_the_committed_series(): """Differential against the model itself: for the committed yoy_growth.yaml, the index build_model_index reads from the model's own diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 4d318c094e..23374d73ab 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1737,13 +1737,15 @@ class LevelSeriesSpec: ), ) # A footnote marker ending a level table's title, unit or label. -LEVEL_FOOTNOTE_RE = re.compile(r"(?<=[a-z)\]])(?:\d+|[⁰¹²³⁴-⁹]+)$", re.IGNORECASE) +LEVEL_FOOTNOTE_RE = re.compile( + r"(?<=[a-z)\]])(?:\d+|[\u2070\u00b9\u00b2\u00b3\u2074-\u2079]+)$", re.IGNORECASE +) # A statement of the year the government has set the rate up until, with any # negation before it and the whole token after it, and a range such as # "2027 - 28" with its spaces. SET_BY_GOVERNMENT_RE = re.compile( r"(?:\b(not|never)\s+(?:yet\s+)?(?:been\s+)?)?" - r"set by the government up until (\S+(?:\s*[-–—]\s*\S+)?)" + r"set by the government up until (\S+(?:\s*[-\u2013\u2014]\s*\S+)?)" ) OBR_ESTIMATES_RE = re.compile(r"figures beyond (\d{4}) are obr estimates") From 48de2a96aed1e982ac65668f80c700d0507da6c8 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sat, 31 Oct 2026 18:37:12 -0400 Subject: [PATCH 15/22] Give the direct main test the synthetic national living wage model MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit test_main_takes_the_release_from_the_economy_contents_sheet calls main without run_main, so it read the parameter file's £13.21 for 2027 against the test table's £12.00 and the importer refused the table. The synthetic_nlw_model context manager now patches the index builder and the anchor for both. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 33 +++++++++++-------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 6900f3c594..c0cef67dbb 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -1,5 +1,6 @@ import math import os +from contextlib import contextmanager from dataclasses import replace from datetime import date from functools import partial @@ -902,6 +903,21 @@ def write_tree(tmp_path): return yoy, gap_dir +@contextmanager +def synthetic_nlw_model(): + """Run main on the test files with the national living wage's index from + create_economic_assumption_indices on the test file alone, and its + anchor from the rate the test table's note sets. The model's other + generation steps cannot read the fixture's short growth series (its rpi + starts in 2024).""" + with pytest.MonkeyPatch.context() as patch: + patch.setattr(import_obr_forecasts, "build_model_index", fixture_index) + patch.setattr( + import_obr_forecasts, "read_level_anchor", lambda spec: CLI_NLW_ANCHOR + ) + yield + + def run_main( tmp_path, workbook, *extra, receipts=None, previous_receipts=None, previous=True ): @@ -911,15 +927,7 @@ def run_main( receipts_path.write_bytes(receipts or make_receipts_xlsx()) previous_path = tmp_path / "previous_receipts.xlsx" previous_path.write_bytes(previous_receipts or make_previous_receipts_xlsx()) - # The model's generation steps cannot read the fixture's short growth - # series (its rpi starts in 2024), so main takes the national living - # wage's index from create_economic_assumption_indices on the fixture - # alone, and its anchor from the rate the test table's note sets. - with pytest.MonkeyPatch.context() as patch: - patch.setattr(import_obr_forecasts, "build_model_index", fixture_index) - patch.setattr( - import_obr_forecasts, "read_level_anchor", lambda spec: CLI_NLW_ANCHOR - ) + with synthetic_nlw_model(): return main( [ "--file", @@ -2036,8 +2044,8 @@ def test_main_takes_the_release_from_the_economy_contents_sheet(tmp_path): receipts.write_bytes(make_receipts_xlsx()) previous = tmp_path / "previous_receipts.xlsx" previous.write_bytes(make_previous_receipts_xlsx()) - assert ( - main( + with synthetic_nlw_model(): + code = main( [ "--file", str(economy), @@ -2053,8 +2061,7 @@ def test_main_takes_the_release_from_the_economy_contents_sheet(tmp_path): str(yoy), ] ) - == 0 - ) + assert code == 0 assert "OBR EFO March 2026" in yoy.read_text() From 22fb1e21a35438e33becba248f81176b2f7ad71f Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 01:26:44 -0400 Subject: [PATCH 16/22] Project a table's first year after the anchor; read anchor dates as core does A table that begins after the parameter file's last national living wage rate has no ratio into its first year. penny_preserving_growth now carries the file's rate there through the growth the importer keeps and refuses the table, naming the year, the OBR's level, the projected rate and the file's rate, where that projection rounds to another penny. Before, the penny check read growth the table never had and raised KeyError. read_level_anchor loads non_apprentice.yaml with policyengine-core's loader and reads its dated values through core's Parameter. dated_values keys each by the day core's instant() reads, so a quoted '2028-04-01' beside unquoted dates, a 2028-10 key and a value marked expected anchor where core does. It refuses two spellings of one day. nearest_units shares core's interval-first rounding between the penny check and the refusal's text. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 144 +++++++++++++++++- policyengine_uk/utils/import_obr_forecasts.py | 111 ++++++++++---- 2 files changed, 227 insertions(+), 28 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index c0cef67dbb..829899e4f6 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -2,7 +2,7 @@ import os from contextlib import contextmanager from dataclasses import replace -from datetime import date +from datetime import date, timedelta from functools import partial from pathlib import Path from io import BytesIO @@ -3171,6 +3171,95 @@ def test_the_anchor_is_the_last_national_living_wage_rate_in_its_file(): assert amount.values_list[0].value == anchor.rate +def nlw_parameter_file(tmp_path, monkeypatch, entry): + """Copy non_apprentice.yaml under ``tmp_path`` with ``entry``, YAML text, + added after the last dated value of its last bracket, point the + importer's repo root there, and return the copy's text.""" + text = (MINIMUM_WAGE_DIR / "non_apprentice.yaml").read_text() + assert text.count("\n\nmetadata:") == 1 + text = text.replace("\n\nmetadata:", f"\n{entry}\nmetadata:") + path = tmp_path / MINIMUM_WAGE_DIR.relative_to(MINIMUM_WAGE_DIR.parents[4]) + path.mkdir(parents=True, exist_ok=True) + (path / "non_apprentice.yaml").write_text(text) + monkeypatch.setattr(import_obr_forecasts, "get_repo_root", lambda: tmp_path) + return text + + +@pytest.mark.parametrize( + "entry, instant, rate", + [ + (" '2028-04-01':\n value: 13.51\n", date(2028, 4, 1), 13.51), + (" 2028-10:\n value: 13.6\n", date(2028, 10, 1), 13.6), + (" 2028-04-01: expected\n", date(2027, 4, 1), 13.21), + ], + ids=["quoted_date", "year_and_month", "expected_value"], +) +def test_the_anchor_is_the_last_rate_core_reads( + tmp_path, monkeypatch, entry, instant, rate +): + """An entry after the file's £13.21 from 2027-04-01: £13.51 keyed by the + quoted date '2028-04-01', which yaml.safe_load reads as text beside the + other keys' dates; £13.60 keyed by 2028-10, which policyengine-core reads + as 1 October 2028; or a 2028 value marked expected, which core drops. In + each case the importer anchors on the rate core reads from the same + file, from the same day, and core reads no later rate.""" + text = nlw_parameter_file(tmp_path, monkeypatch, entry) + assert read_level_anchor(NLW_SPEC) == LevelAnchor(instant, rate, NLW_UPRATING) + tree = ParameterNode("", data={"wage": yaml.load(text, Loader=config.Loader)}) + amount = tree.wage.brackets[-1].amount + assert amount(instant.isoformat()) == rate + assert amount((instant - timedelta(days=1)).isoformat()) != rate + assert amount(date(2074, 12, 31).isoformat()) == rate + + +def test_the_quoted_date_mixes_key_types_for_yaml_safe_load(tmp_path, monkeypatch): + """The quoted '2028-04-01' gives yaml.safe_load a text key beside the date + keys of the file's other entries, which max() cannot compare; core's + loader reads every key as text.""" + text = nlw_parameter_file( + tmp_path, monkeypatch, " '2028-04-01':\n value: 13.51\n" + ) + keys = yaml.safe_load(text)["brackets"][-1]["amount"]["values"] + assert {type(key) for key in keys} == {date, str} + with pytest.raises(TypeError): + max(keys) + core = yaml.load(text, Loader=config.Loader)["brackets"][-1]["amount"]["values"] + assert {type(key) for key in core} == {str} + + +def test_a_day_keyed_twice_is_refused(tmp_path, monkeypatch): + """A quoted '2027-04-01' beside the unquoted 2027-04-01 the file holds: + policyengine-core's loader refuses the duplicate key, and the importer, + which loads the file with it, refuses it too.""" + text = nlw_parameter_file( + tmp_path, monkeypatch, " '2027-04-01':\n value: 13.22\n" + ) + for read in ( + lambda: yaml.load(text, Loader=config.Loader), + lambda: read_level_anchor(NLW_SPEC), + ): + with pytest.raises(yaml.YAMLError, match="duplicate key '2027-04-01'"): + read() + + +def test_a_day_spelled_twice_is_refused(tmp_path, monkeypatch): + """2028-04 and 2028-04-01 after the file's last rate: two keys that + policyengine-core reads as two values from 1 April 2028. The importer + refuses the file rather than choose between them.""" + text = nlw_parameter_file( + tmp_path, + monkeypatch, + " 2028-04:\n value: 13.5\n 2028-04-01:\n value: 13.51\n", + ) + tree = ParameterNode("", data={"wage": yaml.load(text, Loader=config.Loader)}) + instants = [ + entry.instant_str for entry in tree.wage.brackets[-1].amount.values_list + ] + assert instants[:2] == ["2028-04-01", "2028-04"] + with pytest.raises(ValueError, match="dates two values on 2028-04-01"): + read_level_anchor(NLW_SPEC) + + def test_a_table_level_the_file_rounds_to_its_rate_projects_from_the_file(): """£13.210001 for 2027, the year the parameter file sets £13.21, and an estimate of £13.595001 for 2028. From the table's unrounded 2027 level @@ -3200,6 +3289,59 @@ def test_a_table_that_disagrees_with_the_rate_set_is_refused(): projected_from_committed(edit_real_table_1_13({"J5": 13.22})) +def test_a_table_that_starts_after_the_files_rate_projects_its_first_year(): + """The OBR publishes £13.51 for 2028 and £13.85 for 2029, from the year + after the parameter file's £13.21 for 2027. With no 2027 level, the + importer has no ratio into 2028: the model carries £13.21 to 2028 + through the growth the importer keeps, 0.022984, and projects £13.51, + the OBR's penny. The importer writes the ratio into 2029, and the model + projects £13.85.""" + workbook = nlw_workbook(nlw_rows((13.51, 13.85), 2028, 2028)) + assert compute_level_growth(workbook)[NLW_PATH].values == {2029: 0.025167} + values, wages = projected_from_committed(workbook) + assert values[2028] == OCTOBER_2026_NLW_GROWTH[2028] + assert values[2029] == 0.025167 + assert [wages[2028], wages[2029]] == pytest.approx([13.51, 13.85], abs=1e-9) + + +def test_a_first_level_the_files_growth_misses_is_refused(): + """£13.55 for 2028, the table's first year, after the parameter file's + £13.21 for 2027: the growth the importer keeps for 2028 takes £13.21 to + £13.51, and no earlier level in the table sets a growth into 2028 that + could reach £13.55. The importer refuses the table and names the year, + the OBR's level, the projected rate and the file's rate.""" + workbook = nlw_workbook(nlw_rows((13.55, 13.85), 2028, 2028)) + with pytest.raises( + ValueError, + match=r"^national_living_wage: the OBR publishes Table 1\.13 from 2028, " + r"after 2027, the year of the last rate " + r"gov\.hmrc\.minimum_wage\.non_apprentice holds: 13\.21 from " + r"2027-04-01\. The model carries that rate to 2028 through the growth " + r"the importer keeps and projects 13\.51, but the OBR publishes 13\.55, " + r"which rounds to 13\.55 to the nearest 0\.01$", + ): + projected_from_committed(workbook) + + +def test_a_table_two_years_after_the_files_rate_projects_from_it(): + """The OBR publishes £13.85 for 2029 and £14.23 for 2030, from two years + after the parameter file's £13.21 for 2027. The growth the importer + keeps for 2028 and 2029 takes £13.21 to £13.85, the OBR's penny. The + ratio into 2030, 0.027437, would take the model to £14.24, so the + importer writes the growth from the model's 2029 level to £14.23, + 0.027073, and the model projects £14.23.""" + workbook = nlw_workbook(nlw_rows((13.85, 14.23), 2029, 2029)) + assert compute_level_growth(workbook)[NLW_PATH].values == {2030: 0.027437} + values, wages = projected_from_committed(workbook) + assert {year: values[year] for year in (2028, 2029)} == { + year: OCTOBER_2026_NLW_GROWTH[year] for year in (2028, 2029) + } + assert values[2030] == 0.027073 + assert [wages[2029], wages[2030]] == pytest.approx([13.85, 14.23], abs=1e-9) + ratio = model_wages({**values, 2030: 0.027437}, 2027, 13.21) + assert ratio[2030] == pytest.approx(14.24, abs=1e-9) + + def nlw_comments(rendered): """The national_living_wage series' comment lines in a rendered file.""" block = rendered[rendered.index(" national_living_wage:") :] diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 23374d73ab..b4d6801537 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1975,17 +1975,40 @@ def compute_level_growth( return result +def dated_values(name: str, values: dict) -> dict[date, float]: + """Return a parameter's values by the day each takes effect. + policyengine-core's Parameter reads ``values`` (keys as text, as core's + YAML loader gives them) and drops any value marked expected; each key + then becomes the day core's instant() reads from it, so 2028-04-01, + quoted or not, and 2028-04 both name 1 April 2028. It refuses two keys + that name the same day.""" + from policyengine_core.parameters import Parameter + from policyengine_core.periods import instant + + dated = {} + for entry in Parameter(name, {"values": values}).values_list: + day = instant(entry.instant_str).date + if day in dated: + raise ValueError(f"{name} dates two values on {day}") + dated[day] = entry.value + return dated + + def read_level_anchor(spec: LevelSeriesSpec) -> LevelAnchor: """Read the last dated value of ``spec.anchor``'s last bracket's amount, and its uprating metadata, from its file in the model's parameter tree. - It refuses an amount that the series' index does not uprate to the - nearest ``spec.rounding``.""" + It loads the file with policyengine-core's YAML loader, which refuses a + key the file repeats, and takes the latest day dated_values reads, so + it anchors where core does. It refuses an amount that the series' index + does not uprate to the nearest ``spec.rounding``.""" + from policyengine_core.parameters import config + path = ( get_repo_root() / "policyengine_uk/parameters" / Path(*spec.anchor.split(".")) ) - amount = yaml.safe_load(path.with_suffix(".yaml").read_text())["brackets"][-1][ - "amount" - ] + amount = yaml.load(path.with_suffix(".yaml").read_text(), Loader=config.Loader)[ + "brackets" + ][-1]["amount"] uprating = amount["metadata"]["uprating"] expected = { "parameter": "gov.economic_assumptions.indices.obr." + ".".join(spec.path), @@ -1996,13 +2019,9 @@ def read_level_anchor(spec: LevelSeriesSpec) -> LevelAnchor: f"{spec.anchor}'s last bracket must uprate by {expected['parameter']} " f"to the nearest {spec.rounding}; its metadata gives {uprating}" ) - instant = max(amount["values"]) - value = amount["values"][instant] - return LevelAnchor( - instant=date.fromisoformat(str(instant)), - rate=float(value["value"] if isinstance(value, dict) else value), - uprating=uprating, - ) + values = dated_values(f"{spec.anchor}'s last bracket", amount["values"]) + instant = max(values) + return LevelAnchor(instant=instant, rate=float(values[instant]), uprating=uprating) def compound_model_index(values: dict, last_year: int) -> dict[int, float]: @@ -2020,16 +2039,35 @@ def compound_model_index(values: dict, last_year: int) -> dict[int, float]: return index +def nearest_units(amount: float, rounding: float) -> float: + """Return ``amount`` as a whole number of ``rounding``, as + uprate_parameters (policyengine-core) rounds a projected rate to the + nearest interval: it divides by the interval first and rounds the + quotient with Python's round(), which takes a half to the even number. + £13.525 over £0.01 gives exactly 1,352.5 as a float, so £13.525 becomes + 1,352 pennies. Infinity if the division overflows.""" + units = amount / rounding + return round(units) if math.isfinite(units) else math.inf + + +def level_text(level: float, rounding: float) -> str: + """Write ``level`` to the nearest ``rounding`` as the model rounds it + (nearest_units), with the interval's decimal places: "13.52" for + £13.525. If the division overflows, it formats the level itself to + those places.""" + places = max(0, -Decimal(repr(rounding)).as_tuple().exponent) + units = nearest_units(level, rounding) + return f"{units * rounding if math.isfinite(units) else level:.{places}f}" + + def projected_rate_units( index: dict[int, float], start: int, rate: float, year: int, rounding: float ) -> float: """Return the rate uprate_parameters (policyengine-core) projects for ``year`` from ``rate``, in force in ``start``, as a whole number of - ``rounding``: the rate times the index's growth since, over - ``rounding``, rounded to the nearest whole number. Infinity if that - overflows.""" - projected = rate * (index[year] / index[start]) / rounding - return round(projected) if math.isfinite(projected) else math.inf + ``rounding``: the rate times the index's growth since, rounded to the + nearest ``rounding`` (nearest_units). Infinity if that overflows.""" + return nearest_units(rate * (index[year] / index[start]), rounding) def _level_growth_series(series: LevelGrowth, kept: dict) -> dict[str, float]: @@ -2067,20 +2105,24 @@ def penny_preserving_growth( rate from the anchor's rate (projected_rate_units). The importer refuses a table whose level in the anchor's year differs from the anchor's rate at that rounding, and one whose index starts after the anchor's year. - Each later year keeps its growth if the projection rounds to the table's - level. Otherwise the importer takes the growth, to six decimals, from - the level the index carries the rate to in the year before to the - table's level, or else the nearest six-decimal growth to that which does - round to the table's level. It refuses a year that no six-decimal growth - can reproduce. + Where the OBR publishes the table's first level for a year after the + anchor's, the importer has no ratio into that year: the model carries + the anchor's rate there through the growth the importer keeps + (``kept``), and the importer refuses the table if that projection rounds + to another level than the OBR's. Each later year keeps its growth if the + projection rounds to the table's level. Otherwise the importer takes the + growth, to six decimals, from the level the index carries the rate to in + the year before to the table's level, or else the nearest six-decimal + growth to that which does round to the table's level. It refuses a year + that no six-decimal growth can reproduce. """ name = ".".join(path) line = series.line rounding = series.spec.rounding start, rate, last = anchor.instant.year, anchor.rate, max(line.levels) - if start in line.levels and round(line.levels[start] / rounding) != round( - rate / rounding - ): + if start in line.levels and nearest_units( + line.levels[start], rounding + ) != nearest_units(rate, rounding): raise ValueError( f"{name}: the OBR's {start} level, {line.levels[start]!r}, differs " f"to the nearest {rounding} from {rate!r}, the rate " @@ -2110,7 +2152,22 @@ def penny_preserving_growth( ) if not math.isfinite(line.levels[year] / rounding): raise refusal - target = round(line.levels[year] / rounding) + target = nearest_units(line.levels[year], rounding) + if year not in values: + projected = projected_rate_units(index, start, rate, year, rounding) + if projected != target: + raise ValueError( + f"{name}: the OBR publishes Table {line.table} from {year}, " + f"after {start}, the year of the last rate " + f"{series.spec.anchor} holds: {rate!r} from " + f"{anchor.instant}. The model carries that rate to {year} " + "through the growth the importer keeps and projects " + f"{level_text(projected * rounding, rounding)}, but the OBR " + f"publishes {line.levels[year]!r}, which rounds to " + f"{level_text(line.levels[year], rounding)} to the nearest " + f"{rounding}" + ) + continue def units(micros: int) -> float: try: From 58dce2850e88680e3cccec62e11774504b103196 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 01:31:08 -0400 Subject: [PATCH 17/22] Write the comments' levels to the penny the model projects MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit render_level_values formatted each comment's levels with :.2f, so the OBR's £13.525 read £13.53 where the model projects £13.52. The comments now go through level_text, which writes the penny nearest_units gives, the rounding the penny check uses: core divides by the interval and round() takes an exact half to the even penny. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 42 +++++++++++++++++++ policyengine_uk/utils/import_obr_forecasts.py | 10 +++-- 2 files changed, 49 insertions(+), 3 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 829899e4f6..51904d6b62 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -47,7 +47,9 @@ format_growth, get_yoy_growth_path, infer_workbook_release, + level_text, main, + nearest_units, projected_rate_units, read_calendar_values, read_level_anchor, @@ -3349,6 +3351,46 @@ def nlw_comments(rendered): return [line.strip() for line in block.split("\n") if line.strip().startswith("#")] +def test_a_comment_names_the_penny_the_model_projects(): + """The OBR's £13.525 for 2028 after £13.21 for 2027. policyengine-core + divides by the penny first, which gives exactly 1,352.5 pennies, and + Python's round() takes the half to the even 1,352, so the model + projects £13.52. The comment on the estimate names £13.52, not the + £13.53 that formatting the level to two places gives.""" + workbook = nlw_workbook(nlw_rows((13.21, 13.525), 2027, 2027)) + rendered = render_level_series_yaml( + get_yoy_growth_path().read_text(), + compute_level_growth(workbook), + ("October", 2026), + ) + assert (13.525 / 0.01, f"{13.525:.2f}") == (1352.5, "13.53") + assert nlw_comments(rendered)[-2] == ( + "# The OBR estimates the rates for 2028: £13.21 in 2027 to £13.52 in " + '2028 (OBR EFO October 2026, economy Table 1.13 "National Living Wage ' + '(NLW)")' + ) + assert wages_in_the_model(rendered)[2028] == pytest.approx(13.52, abs=1e-9) + + +@pytest.mark.parametrize( + "level, pennies, text", + [ + (13.525, 1352, "13.52"), + (13.535, 1354, "13.54"), + (2.675, 268, "2.68"), + (13.5250001, 1353, "13.53"), + (1e-100, 0, "0.00"), + ], +) +def test_level_text_writes_the_penny_check_s_penny(level, pennies, text): + """level_text writes the penny nearest_units, the penny check's + rounding, gives. Over £0.01, £13.525, £13.535 and £2.675 each give a + float ending in exactly .5, which round() takes to the even penny: + formatting to two places gives £13.53 and £2.67 instead.""" + assert nearest_units(level, 0.01) == pennies + assert level_text(level, 0.01) == text + + def test_a_note_a_year_after_the_files_rate_leaves_the_file_to_anchor(): """The real table with a note that the government has set the rate up until 2028, a year after the parameter file's last rate, and £13.515001 diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index b4d6801537..dcd3e957f8 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -2353,7 +2353,8 @@ def render_level_values( ) -> list[str]: """Render a level series' values entries: the earlier years it keeps, then the growth into the rates the government has set, then into the - OBR's estimates, then the average_earnings path.""" + OBR's estimates, then the average_earnings path. Each comment writes the + OBR's levels to the penny the model rounds a rate to (level_text).""" first_year = min(growth.values) kept: list[str] = [] pending: list[str] = [] @@ -2389,9 +2390,12 @@ def render_level_values( continue span = f"{years[0]}" if len(years) == 1 else f"{years[0]}-{years[-1]}" start, end = years[0] - 1, years[-1] + first, last = ( + level_text(published.levels[year], growth.spec.rounding) + for year in (start, end) + ) lines.append( - f"{pad}# {kind} {span}: £{published.levels[start]:.2f} in {start} " - f"to £{published.levels[end]:.2f} in {end} ({source})" + f"{pad}# {kind} {span}: £{first} in {start} to £{last} in {end} ({source})" ) lines += [f"{pad}{y}-01-01: {format_growth(growth.values[y])}" for y in years] lines.append( From e7fd50d6e4742b6290912370a980ece36c87f73b Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 01:38:50 -0400 Subject: [PATCH 18/22] Bound the national living wage's index guard to the builder's horizon The render ran #2761's check_index_stays_positive on the finished series, which compounds to 2074 whatever the builder does. On this branch the builder ends the index in 2039, so the importer refused a falling average_earnings tail that the model's index holds above zero. check_level_index_in_model now refuses the model's own index, in every year the builder writes, where it reaches zero (from then the model projects no rate above zero from an earlier year's rate) or infinity (uprate_parameters cannot round the rate it projects). The importer compounds compound_model_index to the same year and still checks it against the model's index. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 108 ++++++++++++++---- policyengine_uk/utils/import_obr_forecasts.py | 32 ++++-- 2 files changed, 110 insertions(+), 30 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 51904d6b62..c2998fbc60 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -4120,25 +4120,82 @@ def penny(level): def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): """Growth through the table keeps the index above zero; the falling - average_earnings tail then takes it to zero, so the importer refuses the - render.""" + average_earnings tail then takes the index the builder writes to zero + in 2031, so the importer refuses the render.""" workbook = nlw_workbook(nlw_rows((10.0, 10.0, 0.0001), 2023, 2023)) render_nlw(workbook=workbook) flat = FLAT_SERIES_FIXTURE.format(key="average_earnings") content = YOY_GROWTH_FIXTURE.replace(flat, flat.replace("0.0200", "-0.5100")) assert content != YOY_GROWTH_FIXTURE with pytest.raises( - ValueError, match="national_living_wage: the index reaches 0.0 in 2031" + ValueError, + match="^national_living_wage: the model's index reaches 0.0 in 2031 at five " + "decimals, so from 2031 the model projects no rate above zero from an " + "earlier year's rate$", ): render_nlw(content, workbook) +def falling_after_2028(content): + """``content`` with average_earnings falling 51% a year after 2028.""" + lines = content.split("\n") + start, end = import_obr_forecasts._series_block(lines, ("obr", "average_earnings")) + first, last = import_obr_forecasts._yaml_block(lines, "values", 4, start, end) + for number in range(first + 1, last): + match = import_obr_forecasts.VALUE_LINE_RE.match(lines[number]) + if match and int(match.group(1)) > 2028: + lines[number] = lines[number].split(":", 1)[0] + ": -0.5100" + falling = "\n".join(lines) + assert falling != content + return falling + + +def test_the_index_guard_ends_where_the_builder_ends_the_index(): + """The committed file with average_earnings falling 51% a year after + 2028, and the OBR's £13.21 for 2027 and £13.51 for 2028, a ratio of + 0.02271. Compounded to 2074, the finished series' index reaches zero in + 2045. Where the + model's builder ends the index before 2045 (in 2039 on this branch), + the model's index stays above zero: the importer writes the series, and + the model projects £13.51 for 2028. Where the builder runs the index + to 2045 or later, the importer refuses the series and names 2045, the + year the model's own index reaches zero.""" + content = falling_after_2028(get_yoy_growth_path().read_text()) + growth = compute_level_growth(nlw_workbook(nlw_rows((13.21, 13.51), 2027, 2027))) + render = partial(render_level_series_yaml, content, growth, ("October", 2026)) + with pytest.MonkeyPatch.context() as patch: + patch.setattr( + import_obr_forecasts, "check_level_index_in_model", lambda *_: None + ) + unchecked = render() + written = _values(unchecked, NLW_PATH) + assert written[2028] == 0.02271 + to_2074 = compounded(written, 2074) + assert min(year for year, value in to_2074.items() if value <= 0) == 2045 + model = build_model_index(unchecked, NLW_PATH) + assert model == {year: to_2074[year] for year in model} + if max(model) < 2045: + assert all(value > 0 for value in model.values()) + assert render() == unchecked + wages = uprated_wages(model, read_level_anchor(NLW_SPEC)) + assert wages[2028] == pytest.approx(13.51, abs=1e-9) + return + with pytest.raises( + ValueError, + match="^national_living_wage: the model's index reaches 0.0 in 2045 at five " + "decimals, so from 2045 the model projects no rate above zero from an " + "earlier year's rate$", + ): + render() + + def test_a_finished_index_that_overflows_is_refused(): """£0.00001 for 2027 and £1.7e303 for 2028 give finite growth, 1.7e308. In the committed file the kept years take the index above 1 by 2027, so that growth takes it to infinity in 2028: the importer refuses the - table, and check_index_stays_positive refuses the finished series it - would write.""" + table, and the check on the model's own index refuses the finished + series it would write, where uprate_parameters could not round the + 2028 rate.""" workbook = nlw_workbook(nlw_rows((1e-5, 1.7e303), 2027, 2027)) growth = compute_level_growth(workbook)[NLW_PATH] assert math.isfinite(growth.values[2028]) and growth.values[2028] > 1e308 @@ -4155,10 +4212,17 @@ def test_a_finished_index_that_overflows_is_refused(): for year, value in average_earnings_years(committed, 2028).items() }, } + content = yaml.safe_load(committed) + content["obr"]["national_living_wage"]["values"] = finished with pytest.raises( - ValueError, match="national_living_wage: the index reaches inf in 2028" + ValueError, + match="^national_living_wage: the model's index reaches inf in 2028 at five " + "decimals, so policyengine-core's uprate_parameters cannot round the " + "rate it projects for 2028$", ): - import_obr_forecasts.check_index_stays_positive(NLW_PATH, finished) + import_obr_forecasts.check_level_index_in_model( + yaml.safe_dump(content, sort_keys=False), NLW_PATH, growth + ) with pytest.raises( ValueError, match=r"national_living_wage: 2028 growth of 1\.7e\+308 takes the index " @@ -4190,18 +4254,21 @@ def test_growth_past_every_float_is_refused(): # A fall to a thousandth keeps the index above zero through the table; the # falling tail then takes the finished index to zero in 2037. @example(levels=[1.0, 0.001], earnings=-0.51) +# A flat table, then the falling tail: compounded to 2074, the finished +# index reaches zero in 2047, after the year this branch's builder ends it. +@example(levels=[1.0, 1.0], earnings=-0.51) def test_a_written_level_series_keeps_every_index_positive(levels, earnings): """For generated positive levels (2 to 8 years from 2023, drawn from TINY_TO_HUGE) and an average_earnings tail rising 2% or falling 51% a year: the importer refuses the render exactly when the five-decimal index the model compounds through the written series (the years the - file keeps before the table, the table's six-decimal growth, then - average_earnings) reaches zero or infinity by LAST_INDEX_YEAR. It names - the table year where the index first does so, or the later year where - the finished series' index does. When it writes the series, the model's - index equals that index in every year the model builds. The property - leaves the penny check out, as levels this far apart rarely round to a - penny the index can reach.""" + importer keeps before the table, the table's six-decimal growth, then + average_earnings) reaches zero or infinity by the year the builder ends + the index. It names the table year where the index first does so, or + the later year where the model's index does. When it writes the series, + the model's index equals that index in every year the model builds. The + property leaves the penny check out, as levels this far apart rarely + round to a penny the index can reach.""" content = earnings_fixture(earnings) workbook = nlw_workbook(nlw_rows(tuple(levels), 2023, 2023)) values = { @@ -4216,15 +4283,11 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): lambda _, series, *__: series, ) with pytest.MonkeyPatch.context() as unchecked_patch: - for check in ( - "check_level_index", - "check_index_stays_positive", - "check_level_index_in_model", - ): + for check in ("check_level_index", "check_level_index_in_model"): unchecked_patch.setattr(import_obr_forecasts, check, lambda *_: None) unchecked = render_nlw(content, workbook) written = _values(unchecked, NLW_PATH) - finished = compounded(written, import_obr_forecasts.LAST_INDEX_YEAR) + finished = compounded(written, max(fixture_index(unchecked, NLW_PATH))) failed = [ year for year, level in finished.items() @@ -4241,8 +4304,9 @@ def test_a_written_level_series_keeps_every_index_positive(levels, earnings): ) else: event("refused: finished series") - assert f"the index reaches {finished[failed[0]]} in {failed[0]}" in str( - error + assert ( + f"the model's index reaches {finished[failed[0]]} in {failed[0]}" + in str(error) ) return event("written") diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index dcd3e957f8..2f2c6d3158 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -2305,8 +2305,12 @@ def check_level_index_in_model( """Check the index the model builds from ``content``'s ``obr.`` against the importer's. ``build_index(content, path)`` builds it; by default build_model_index does, as the model does. The index must reach - the table's last year, and compound_model_index must compound the same - index from the written series in every year the model's has.""" + the table's last year and stay above zero and finite in every year the + builder writes: from a year where it reaches zero, the model projects + no rate above zero from an earlier year's, and an infinite index stops + uprate_parameters rounding the rates it projects. compound_model_index + must compound the same index from the written series in each of those + years.""" name = ".".join(path) model = (build_index or build_model_index)(content, path) last = max(series.line.levels) @@ -2315,6 +2319,19 @@ def check_level_index_in_model( f"{name}: the model's index ends in {max(model)}, so it cannot " f"project the {last} level" ) + for year, value in sorted(model.items()): + if not (math.isfinite(value) and value > 0): + effect = ( + f"from {year} the model projects no rate above zero from an " + "earlier year's rate" + if math.isfinite(value) + else "policyengine-core's uprate_parameters cannot round the " + f"rate it projects for {year}" + ) + raise ValueError( + f"{name}: the model's index reaches {value} in {year} at five " + f"decimals, so {effect}" + ) node = yaml.safe_load(content)["obr"] for key in path: node = node[key] @@ -2423,11 +2440,11 @@ def render_level_series_yaml( anchor (``anchors``, or by default read_level_anchor), so the model projects each table level after the anchor's year to the table's penny. Years after the table follow average_earnings as ``content`` holds it. - It checks that the finished series' index stays above zero, that the - result differs from ``content`` only in those values and that reference, - and that the index the model builds from the result - (check_level_index_in_model, with ``build_index``) matches the - importer's. + It checks that the result differs from ``content`` only in those values + and that reference. It then builds the model's index from the result + (check_level_index_in_model, with ``build_index``) and checks that the + index reaches the table's last year, stays above zero and finite to the + year the builder ends it, and matches the importer's. """ month, year = release expected = copy.deepcopy(yaml.safe_load(content)) @@ -2486,7 +2503,6 @@ def render_level_series_yaml( for entry_year, value in {**series.values, **tail}.items() }, } - check_index_stays_positive(path, node["values"]) node["metadata"]["reference"][0] = reference written.append((path, series)) From c6c32dd774712ff2cc813f6de3c7b7a8f5c42bb3 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 01:52:12 -0400 Subject: [PATCH 19/22] Test that a table after the rate set keeps every penny A property draws a rate set from 2021 to 2029 and a table the OBR publishes from 1 to 3 years later, with its first level near the penny the model carries the rate to or a penny off it. The importer accepts the table only where the first level rounds to that penny, and the model then projects every later level to its penny. Co-Authored-By: Claude Opus 5.5 --- .../tests/test_import_obr_forecasts.py | 65 +++++++++++++++++++ 1 file changed, 65 insertions(+) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index c2998fbc60..51a9faba85 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -4118,6 +4118,71 @@ def penny(level): assert kind == ("set" if year <= set_until else "estimate"), year +@settings(max_examples=40, deadline=None, derandomize=True) +@given( + set_year=st.integers(2021, 2029), + gap=st.integers(1, 3), + pennies=st.integers(500, 2000), + changes=st.lists(NLW_CHANGES, min_size=1, max_size=6), + miss=st.sampled_from((0, 0, 1, -1)), + jitter=st.floats(min_value=-0.0045, max_value=0.0045, allow_nan=False), +) +def test_a_table_after_the_rate_set_keeps_every_penny( + set_year, gap, pennies, changes, miss, jitter +): + """For generated tables the OBR publishes from 1 to 3 years after the + rate set (£5 to £20, set in a year from 2021 to 2029; then up to 6 + more levels, each changing by -5% to +10%): the model carries the rate + set to the table's first year through the growth the importer keeps. + The test puts the OBR's first level within 0.45p of the penny the model + projects there, or a penny above or below it. The importer accepts the + table only where the first level rounds to that penny, and the model + then projects every later level to its penny too. Otherwise it refuses + the table and names the table's first year.""" + anchor = LevelAnchor(date(set_year, 4, 1), pennies / 100, NLW_UPRATING) + first_year = set_year + gap + kept = { + year: value + for year, value in _values(EARLY_EARNINGS_FIXTURE, NLW_PATH).items() + if year <= first_year + } + # The index to the table's first year depends only on the kept growth; + # a 2039 entry runs it past the table on any builder. + carried = uprated_wages(model_index({**kept, 2039: 0.0}), anchor)[first_year] + levels = [carried + miss / 100 + jitter] + for change in changes: + levels.append(levels[-1] * (1 + change)) + workbook = nlw_workbook(nlw_rows(tuple(levels), first_year, first_year)) + render = partial( + render_level_series_yaml, + EARLY_EARNINGS_FIXTURE, + compute_level_growth(workbook), + ("March", 2026), + {NLW_PATH: anchor}, + fixture_index, + ) + if miss: + event("refused: the first level") + with pytest.raises( + ValueError, + match=f"the OBR publishes Table 1.13 from {first_year}, after {set_year}", + ): + render() + return + try: + rendered = render() + except ValueError as error: + event("refused: a later level") + assert "growth to six decimals" in str(error), error + return + event("written") + wages = uprated_wages(fixture_index(rendered, NLW_PATH), anchor) + for offset, level in enumerate(levels): + assert wages[first_year + offset] == pytest.approx( + round(level / 0.01) * 0.01, abs=1e-9 + ), first_year + offset + + def test_a_tail_that_rounds_the_level_index_to_zero_is_refused(): """Growth through the table keeps the index above zero; the falling average_earnings tail then takes the index the builder writes to zero From 0040cf81af6678597200106292d443638b41ed6c Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 01:56:52 -0400 Subject: [PATCH 20/22] Describe the national living wage import as it now runs The README states the growth as each level over the previous one, less one, rounded to six decimals, names the file the model projects from and the first-year check after it, and lists the refusals the importer now makes: a first level the model carries to another penny, and a model index that reaches zero or infinity in a year it builds. Co-Authored-By: Claude Opus 5.5 --- .../gov/economic_assumptions/README.md | 40 ++++++++++++------- 1 file changed, 25 insertions(+), 15 deletions(-) diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index 0c8589ddee..b3b69f1d1e 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -76,20 +76,27 @@ rates, then `average_earnings`' long-run growth, year for year. HM Treasury costed Budget 2026's Carer's Allowance taper on the OBR's estimates. `utils/import_obr_forecasts.py` writes the series with the other economy -series. It finds the table by its title, reads the level under each year -across the table's one row of years, and writes each year's level over the -year before's, to six decimals. The OBR identifies in its note the last year -the government has set, and the importer labels the growth into those years -and into the OBR's estimates with separate comments. The model projects the -national living wage, the last bracket of -`gov/hmrc/minimum_wage/non_apprentice.yaml`, from that bracket's last rate, -through the index it builds from this series, to the nearest penny. The -importer projects each later year of the table from the same rate in the same -way, and where a year's ratio would leave the model a penny off the OBR's -level, it writes the six-decimal growth that keeps the OBR's penny. It then -runs the model's own generation steps on the file it would write, and checks -that the model's index reaches the table's last year and matches the index -the importer compounded. +series. It finds the table by its title and reads the level under each year +across the table's one row of years. The importer divides each year's level +by the previous year's level, subtracts one, and rounds to six decimals. The +OBR identifies in its note the last year the government has set, and the +importer labels the growth into those years and into the OBR's estimates with +separate comments. + +The model projects the national living wage from the last rate in the last +bracket of `gov/hmrc/minimum_wage/non_apprentice.yaml`. It carries that rate +through the index it builds from this series and rounds each projected rate +to the nearest penny. The importer reads the same rate from the same file, as +policyengine-core reads it, and projects each later year of the table from it +in the same way. Where a year's ratio would leave the model a penny off the +OBR's level, the importer writes the six-decimal growth that keeps the OBR's +penny. Where the OBR publishes no level for the year of that rate or earlier, +the importer keeps the series' earlier growth into the table's first year and +checks the model's penny there against the OBR's. The importer +then runs the model's own generation steps on the file it would write. It +checks that the model's index reaches the table's last year, stays above zero +and finite in every year the model builds it, and matches the index the +importer compounded. The importer stops before writing any file when: @@ -101,7 +108,10 @@ The importer stops before writing any file when: statements name different years; - the OBR's level in the year of the file's last rate differs from that rate at the penny; -- the table's growth takes the model's index to zero or infinity; or +- the OBR publishes no level for the year of that rate or earlier, and the + model carries the rate to another penny in the table's first year; +- the model's index, built from the series the importer would write, reaches + zero or infinity in a year the model builds it; or - no six-decimal growth keeps an estimate's penny. `test_minimum_wage_projection.py` holds the October 2026 table, checks it From b86f0e1d400d08a95f69311cf7748068ef905496 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 05:41:12 -0500 Subject: [PATCH 21/22] Build the falling-tail fixture as a new list test_cached_arrays_not_written_in_place scans every function in the package, tests included, and reads a call with a string literal first, such as content.split("\n"), as an array read from the simulation. falling_after_2028 assigned into the lines that call returned, so the scan failed on it. The helper now builds the edited text from a new list. Co-Authored-By: Claude Opus 5.5 --- policyengine_uk/tests/test_import_obr_forecasts.py | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-) diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index 51a9faba85..a3e517f23f 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -4206,11 +4206,15 @@ def falling_after_2028(content): lines = content.split("\n") start, end = import_obr_forecasts._series_block(lines, ("obr", "average_earnings")) first, last = import_obr_forecasts._yaml_block(lines, "values", 4, start, end) - for number in range(first + 1, last): - match = import_obr_forecasts.VALUE_LINE_RE.match(lines[number]) - if match and int(match.group(1)) > 2028: - lines[number] = lines[number].split(":", 1)[0] + ": -0.5100" - falling = "\n".join(lines) + + def falls(number, line): + match = import_obr_forecasts.VALUE_LINE_RE.match(line) + return first < number < last and match and int(match.group(1)) > 2028 + + falling = "\n".join( + line.split(":", 1)[0] + ": -0.5100" if falls(number, line) else line + for number, line in enumerate(lines) + ) assert falling != content return falling From 17dbd906755e11c3d13d4264aef0ad46fbf12587 Mon Sep 17 00:00:00 2001 From: Max Ghenis Date: Sun, 1 Nov 2026 07:05:25 -0500 Subject: [PATCH 22/22] Name who acts in the National Living Wage importer's notes Co-Authored-By: Claude Opus 5.5 --- .../gov/economic_assumptions/README.md | 6 ++-- .../tests/test_import_obr_forecasts.py | 33 +++++++++---------- policyengine_uk/utils/import_obr_forecasts.py | 4 +-- 3 files changed, 21 insertions(+), 22 deletions(-) diff --git a/policyengine_uk/parameters/gov/economic_assumptions/README.md b/policyengine_uk/parameters/gov/economic_assumptions/README.md index b3b69f1d1e..8cbcce5610 100644 --- a/policyengine_uk/parameters/gov/economic_assumptions/README.md +++ b/policyengine_uk/parameters/gov/economic_assumptions/README.md @@ -86,9 +86,9 @@ separate comments. The model projects the national living wage from the last rate in the last bracket of `gov/hmrc/minimum_wage/non_apprentice.yaml`. It carries that rate through the index it builds from this series and rounds each projected rate -to the nearest penny. The importer reads the same rate from the same file, as -policyengine-core reads it, and projects each later year of the table from it -in the same way. Where a year's ratio would leave the model a penny off the +to the nearest penny. The importer reads that rate through policyengine-core's +loader and Parameter, then projects each later table year with the model's +rounding. Where a year's ratio would leave the model a penny off the OBR's level, the importer writes the six-decimal growth that keeps the OBR's penny. Where the OBR publishes no level for the year of that rate or earlier, the importer keeps the series' earlier growth into the table's first year and diff --git a/policyengine_uk/tests/test_import_obr_forecasts.py b/policyengine_uk/tests/test_import_obr_forecasts.py index a3e517f23f..1a92947ee4 100644 --- a/policyengine_uk/tests/test_import_obr_forecasts.py +++ b/policyengine_uk/tests/test_import_obr_forecasts.py @@ -3330,8 +3330,8 @@ def test_a_table_two_years_after_the_files_rate_projects_from_it(): after the parameter file's £13.21 for 2027. The growth the importer keeps for 2028 and 2029 takes £13.21 to £13.85, the OBR's penny. The ratio into 2030, 0.027437, would take the model to £14.24, so the - importer writes the growth from the model's 2029 level to £14.23, - 0.027073, and the model projects £14.23.""" + importer writes the growth from the model's unrounded 2029 level to + £14.23, 0.027073, and the model projects £14.23.""" workbook = nlw_workbook(nlw_rows((13.85, 14.23), 2029, 2029)) assert compute_level_growth(workbook)[NLW_PATH].values == {2030: 0.027437} values, wages = projected_from_committed(workbook) @@ -3383,10 +3383,10 @@ def test_a_comment_names_the_penny_the_model_projects(): ], ) def test_level_text_writes_the_penny_check_s_penny(level, pennies, text): - """level_text writes the penny nearest_units, the penny check's - rounding, gives. Over £0.01, £13.525, £13.535 and £2.675 each give a - float ending in exactly .5, which round() takes to the even penny: - formatting to two places gives £13.53 and £2.67 instead.""" + """level_text and the penny check both use nearest_units. Dividing + £13.525, £13.535 and £2.675 by £0.01 gives half-penny floats, which + round() takes to the even penny. Formatting £13.525 and £2.675 to two + places gives £13.53 and £2.67.""" assert nearest_units(level, 0.01) == pennies assert level_text(level, 0.01) == text @@ -4130,12 +4130,12 @@ def penny(level): def test_a_table_after_the_rate_set_keeps_every_penny( set_year, gap, pennies, changes, miss, jitter ): - """For generated tables the OBR publishes from 1 to 3 years after the - rate set (£5 to £20, set in a year from 2021 to 2029; then up to 6 - more levels, each changing by -5% to +10%): the model carries the rate - set to the table's first year through the growth the importer keeps. - The test puts the OBR's first level within 0.45p of the penny the model - projects there, or a penny above or below it. The importer accepts the + """The test generates an anchor rate of £5 to £20 in a year from 2021 + to 2029, and a first table level 1 to 3 years later, followed by 1 to + 6 levels changing by -5% to +10%. The model carries the rate to the + table's first year through the growth the importer keeps. The test + puts the first level within 0.45p of the penny the model projects + there, or a penny above or below it. The importer accepts the table only where the first level rounds to that penny, and the model then projects every later level to its penny too. Otherwise it refuses the table and names the table's first year.""" @@ -4220,11 +4220,10 @@ def falls(number, line): def test_the_index_guard_ends_where_the_builder_ends_the_index(): - """The committed file with average_earnings falling 51% a year after - 2028, and the OBR's £13.21 for 2027 and £13.51 for 2028, a ratio of - 0.02271. Compounded to 2074, the finished series' index reaches zero in - 2045. Where the - model's builder ends the index before 2045 (in 2039 on this branch), + """The importer computes growth of 0.02271 from £13.21 for 2027 and + £13.51 for 2028. The test sets average_earnings to fall 51% a year + after 2028. It compounds the finished series through 2074, where its + index reaches zero in 2045. Where the model's builder ends the index before 2045 (in 2039 on this branch), the model's index stays above zero: the importer writes the series, and the model projects £13.51 for 2028. Where the builder runs the index to 2045 or later, the importer refuses the series and names 2045, the diff --git a/policyengine_uk/utils/import_obr_forecasts.py b/policyengine_uk/utils/import_obr_forecasts.py index 2f2c6d3158..ab7f5a5b11 100644 --- a/policyengine_uk/utils/import_obr_forecasts.py +++ b/policyengine_uk/utils/import_obr_forecasts.py @@ -1980,8 +1980,8 @@ def dated_values(name: str, values: dict) -> dict[date, float]: policyengine-core's Parameter reads ``values`` (keys as text, as core's YAML loader gives them) and drops any value marked expected; each key then becomes the day core's instant() reads from it, so 2028-04-01, - quoted or not, and 2028-04 both name 1 April 2028. It refuses two keys - that name the same day.""" + quoted or not, and 2028-04 both name 1 April 2028. dated_values refuses + different keys that name the same day; core retains both.""" from policyengine_core.parameters import Parameter from policyengine_core.periods import instant