Repository navigation
ci: size the cv6xx NOR canvas to the variant it is built for - #2448
Conversation
PR Summary by QodoSize CV6xx NOR release images by firmware variant
AI Description
Diagram
High-Level Assessment
Files changed (1)
|
Code Review by Qodo
1.
|
|
The lite variant is being added in this PR: #2447 |
f615cfd to
b002127
Compare
|
Code review by qodo was updated up to the latest commit b002127 |
b002127 to
c51fd67
Compare
openipc-ai
left a comment
There was a problem hiding this comment.
The sixth argument and the exact-size backstop are the right shape, and the hardcoded 16384 clearly had to go. But the bound the new check enforces is not the one that decides whether the image boots, and I think that has to be settled before this publishes an 8 MiB artifact.
The u-boot this job pairs with the firmware is a HiSilicon boot table followed by a gzip member (offset 40128 for cv610, 84768 for hi3519dv500). Decompressed, the compiled-in default env is:
# boot-hi3516cv610-*-nor.bin
kernaddr=0x50000 kernsize=0x200000 rootaddr=0x250000 rootsize=0x500000
mtdparts=sfc:256k(boot),64k(env),2048k(kernel),5120k(rootfs),7168k@0x50000(firmware),-(rootfs_data)
bootnor=... sf read ${baseaddr} ${kernaddr} ${kernsize}; bootm ${baseaddr}
bootargs=... root=/dev/mtdblock3 ...
# boot-hi3519dv500-*-nor.bin
kernaddr=0xC0000 kernsize=0x600000
mtdparts=512k(boot),256k(env),6144k(kernel),8192k(rootfs),14336k@0xC0000(firmware),-(rootfs_data)
Nothing rewrites that afterwards -- set_allocator only edits the mmz half of bootargs, and no overlay script touches mtdparts or kernsize -- so this is the table the camera boots with.
Measured against what this job currently assembles, and what it would assemble with #2447:
| FIT | kernel part | rootfs at | rootfs part at | blob | firmware part | |
|---|---|---|---|---|---|---|
| cv6xx ultimate (published) | 2709 KiB | 2048 KiB ❌ | +2752 KiB | +2048 KiB ❌ | 10496 KiB | 7168 KiB ❌ |
| cv6xx lite (#2447) | 2496 KiB | 2048 KiB ❌ | +2496 KiB | +2048 KiB ❌ | 7680 KiB | 7168 KiB ❌ |
| hi3519dv500 ultimate (published) | 5586 KiB | 6144 KiB ✅ | +5632 KiB | +6144 KiB ❌ | 14464 KiB | 14336 KiB ❌ |
Two separate disagreements, both pre-existing and neither introduced here. post-image.sh packs the rootfs at the first 64 KiB boundary after the FIT; u-boot expects it at a fixed partition offset -- so root=/dev/mtdblock3 points 704 KiB before the squashfs on cv6xx ultimate and 512 KiB past it on hi3519dv500. And on cv6xx the FIT is larger than the 2048 KiB kernel partition, so sf read ${kernaddr} ${kernsize} hands bootm a truncated image.
general/overlay/usr/sbin/sysupgrade gets both right where this assembler does not: do_update_firmware splits the blob at the FIT boundary and writes each half to its own partition, and check_combined_fits dies rather than write when either half is oversized. So an over-large combined image fails safe over the air -- it only mis-lands on a fresh whole-chip flash of exactly the .bin this job publishes, which is the artifact #2447 is asking for on parts nobody can reach with a programmer twice.
I am requesting changes on the bound, not on the parameterisation. Everything else here should stay. If splitting the blob the way sysupgrade does is too much for one PR, checking fw_kib against the firmware partition size (7168 / 14336) rather than nor - off at least stops this publishing something that overruns into rootfs_data.
One process note: "Hardware tested on: not applicable -- CI machinery that assembles a release artifact" is a stretch against CLAUDE.md, which exempts "CI machinery that only selects, lints or tests". This produces the bytes a user writes to a chip. lint-workflow-shell.py and its self-test do pass here, for what that covers.
c51fd67 to
a8ff89e
Compare
|
Rewritten in Consequence, stated in the description: cv6xx ultimate on master now fails the FIT check and hi3519dv500 ultimate the rootfs check, which is the same disagreement the published images already carry, reported instead of published. Hardware section is no longer "not applicable": the function from this branch, run against the #2447 build and the published |
openipc-ai
left a comment
There was a problem hiding this comment.
The rewrite is the right one, and it is more than I asked for -- splitting at the FIT boundary and placing each half where the env looks is exactly what do_update_firmware does, and I checked the arithmetic lands where it should: 320 + 2048 = 2368 KiB = 0x250000 on cv6xx, 768 + 6144 = 6912 KiB on dv500, both matching the tables. bytes_used really is the little-endian u64 at superblock offset 40, the FDT total size really is the big-endian word at byte 4, and the four bounds are each the partition rather than the chip. My inline points are all addressed; the local line covers every scratch variable.
One thing I got wrong by omission in the first review, and it now blocks: the consequence is not "reported instead of published", it is "nothing is published".
- the
Preparestep declares noshell:, so it runs under Actions' defaultbash -e create_hisireturns 1 as a plain command inside aforloop, soset -eends the stepUploadcarries noif:, so it defaults tosuccess()and is skipped
And the ordering decides the blast radius. Sigmastar and Allwinner assemble at line 238 and Ingenic at 243, both before the cv6xx loop at 257. Their .bin files are sitting in target/ when the first cv6xx ultimate call fails; the dv500 loop at 270 never runs at all; and the upload at 275 never fires. So on the first scheduled assembly after this merges, the image release gets no new artifacts for any SoC -- not two withheld.
Refusing to publish a broken image is right. Ending every other SoC's publication to do it is not, particularly when the two targets being refused are already broken on master, so this is not a regression being caught but a standing disagreement being surfaced.
Report and skip would do it: ::error:: on the target, return 0, and let the rest of the run finish. If the run should still go red -- and it probably should, or nobody will look -- collect the failures instead of returning out of the loop, exit "$rc" at the end of the step, and give Upload an if: always() so what did assemble still ships.
Nothing else outstanding from me. lint-workflow-shell.py and its self-test pass on a8ff89e0.
a8ff89e to
42c7578
Compare
|
Right, and thank you for tracing the blast radius rather than the symptom. |
42c7578 to
cd896b7
Compare
cd896b7 to
fdb2f32
Compare
openipc-ai
left a comment
There was a problem hiding this comment.
The collect-and-continue shape is right, and rm -rf output at function entry is a catch I missed -- the old code only cleaned up on the success path, so a refused call left its extracted blob for the next one. Every refusal now gets its turn, the step still ends red, and what assembled still ships.
One thing left, and it is the same class of harm the PR exists to prevent.
always() does not mean "on failure". It means on failure and on cancellation. create_hisi builds $release in four stages -- erased canvas, u-boot, FIT, rootfs -- so a run cancelled between them leaves a partial .bin in target/: an 0xff canvas carrying a bootloader and no firmware. With if: always() the upload runs anyway and that lands in the image release as a flashable asset, next to the good ones and indistinguishable from them until someone writes it to a chip.
if: ${{ !cancelled() }} keeps everything you wanted from always() -- a refused image still fails the step, the images that did assemble still publish -- and lets a cancellation actually cancel. The ${{ }} is not optional there: if: !cancelled() on its own is invalid YAML, because a leading ! is a tag indicator.
Not a request, just so it is known rather than rediscovered: a wget failure still takes the return 0 path, so a download that fails is skipped silently and does not set hisi_failed. That predates this PR and I would leave it alone here.
Housekeeping while I was in it: I rebased this onto master as fdb2f321, since #2447 landed and the required checks are strict. Clean rebase, patch byte-identical, authorship unchanged. Its workflow runs were sitting at action_required -- fork PR -- so I approved them; this is the first CI this branch has had.
create_hisi laid u-boot at 0 and the combined firmware.bin verbatim at the
firmware offset, on a canvas hardcoded to 16384 KiB. Two things were wrong with
that, one new and one old.
The new one: a lite variant is flashed to an 8 MiB part, and an image whose
tail runs past the end of the chip cannot be written to one. The part size is
now the sixth argument, defaulting to 16384 so every existing caller keeps its
behaviour, and the cv6xx loop emits both variants at the size each is built
for.
The old one: the boot images this job pairs with the firmware carry their
partition table in a compiled-in env, and nothing rewrites it later.
Decompressed from boot-hi3516cv610-*-nor.bin (a HiSilicon boot table with a
gzip'd u-boot behind it):
cv6xx 256k(boot),64k(env),2048k(kernel),5120k(rootfs),7168k@0x50000(firmware),-(rootfs_data)
bootnor: sf read ${kernaddr}=0x50000 ${kernsize}=0x200000; root=/dev/mtdblock3
dv500 512k(boot),256k(env),6144k(kernel),8192k(rootfs),14336k@0xC0000(firmware),-(rootfs_data)
So u-boot reads the kernel as one fixed slot and mounts root at a fixed offset
behind it. firmware.bin is the FIT with the squashfs packed at the next 64 KiB
boundary (board post-image.sh), which only agrees with that table when the FIT
happens to end exactly at the slot. On every published cv6xx and dv500 image it
does not: root=/dev/mtdblockN points 704 KiB before the squashfs on cv6xx
ultimate and 512 KiB past it on dv500, and the cv6xx FIT is larger than its
2048 KiB slot, so `sf read` hands bootm a truncated image. sysupgrade already
gets this right (do_update_firmware splits the blob at the FIT boundary and
writes each half to its partition); a fresh whole-chip flash of the release
.bin did not.
Do what sysupgrade does: split at the FIT boundary (the FIT's total size is the
big-endian word at byte 4; the squashfs length is bytes_used in its
superblock), put the FIT at the firmware offset and the rootfs at the rootfs
partition, and measure each piece against the partition it goes into -- u-boot
against the boot partition (the env sits behind it, so the firmware offset is
the wrong bound), the FIT against the kernel slot, the rootfs against the
rootfs slot, the firmware partition against the part. Each failure prints a
::error:: and fails the step rather than uploading something that boots a
truncated kernel or overwrites rootfs_data. The exact-size check on the
assembled canvas stays as the backstop.
The layout table is a case on $soc inside the job; adding a family means
adding its u-boot's env line there. With this in place the cv6xx ultimate image
on master fails the FIT check (2709 KiB in 2048) and dv500 ultimate fails the
rootfs check (8832 KiB in 8192) -- both are the same table disagreement, now
reported instead of published. OpenIPC#2447 fits both cv6xx slots.
A refusal must not take the other SoCs' images with it. The step runs under
Actions' default `bash -e`, and the Sigmastar, Allwinner and Ingenic images are
assembled before the HiSilicon loops, so a bare `return 1` there would end the
step with their .bin files in target/ and the upload never reached. The
HiSilicon calls collect their failures instead, the step exits 1 after the last
one so the run is red, and Upload runs on `always()` so what did assemble
ships.
always() on the upload step covers failure, which is what the refusal path needs, but it also covers cancellation. create_hisi writes each release in four passes -- erased canvas, u-boot, FIT, rootfs -- so a run cancelled between them leaves a .bin that is a bootloader on an otherwise blank part, and always() publishes it to the image release looking exactly like a finished one. That is the failure this whole change exists to prevent. !cancelled() keeps both properties that mattered: a refused image still fails the step, and the images that did assemble still ship.
fdb2f32 to
fa47769
Compare
openipc-ai
left a comment
There was a problem hiding this comment.
Approving, with one disclosure: I pushed a commit onto this branch rather than hand it back for a single word.
fa47769a changes the upload condition to if: ${{ !cancelled() }} and moves the reasoning into the comment above the step. Nothing else in it. Your commit 85d943ed is untouched and still carries the work; I added mine on top so the change is attributable and one revert away if you disagree with it.
Why I did it rather than ask: master had moved again (#2454), the required checks are strict, so this needed another rebase and another full CI cycle no matter what. Folding the one-word fix into that cycle cost nothing and meant not merging a defect I had just written a review about. If you would rather it were yours, squash it into 85d943ed and force-push -- I will re-approve.
Everything else here is settled and I resolved the threads with what each turned into. For the record, what this PR ends up doing: it splits the combined blob at the FIT boundary, puts each half where the cv610 and dv500 envs actually look, refuses any image whose halves do not fit their slots, and keeps publishing the ones that do. cv6xx ultimate and hi3519dv500 will start failing that refusal on the first run -- both are genuinely broken against their own partition tables today, and the numbers are in my earlier review.
CI on fa47769a -- its fork runs needed approving again, so I did that too.
The bounds checks #2448 added ran for the first time on 2026-09-19 and refused six images: hi3516cv6xx ultimate on all four DDR binnings, and hi3519dv500 ultimate on both. That is not a regression #2448 introduced -- it is the table disagreement the published images already carried, now reported instead of shipped -- but it leaves the job red on every run, and re-running against a fresh nightly does not clear it. That run assembled from nightly-20260918-49908b5, because the build had failed and the manifest never advanced, so cv6xx reported a FIT overflow from a blob predating #2447's kernel trim. Measured against the current nightly with create_hisi's own arithmetic, the error changes but does not go away: hi3516cv6xx_lite FIT 2048/2048 ok rootfs 5088/5120 ok hi3516cv6xx_ultimate FIT 2048/2048 ok rootfs 7096/5120 over by 1976 KiB hi3519dv500_ultimate FIT 5632/6144 ok rootfs 8820/8192 over by 628 KiB The cause is that there is one boot binary per DDR binning, not one per part size, and its compiled-in mtdparts is the same on an 8 MiB part and a 16 MiB one. Decompressing the gzip member inside the shipped u-boots (offset 40128 for cv610, 84768 for hi3519dv500) gives: cv610/cv608 256k(boot),64k(env),2048k(kernel),5120k(rootfs),7168k@0x50000(firmware) hi3519dv500 512k(boot),256k(env),6144k(kernel),8192k(rootfs),14336k@0xC0000(firmware) The cv6xx ultimate blob is 9152 KiB and the dv500 one 14464 KiB -- each larger than the whole firmware partition, never mind the rootfs slot inside it. FLASH_SIZE="16" buys them nothing, because the rest of the chip is rootfs_data. No trim closes a 1976 KiB gap, so until a u-boot ships whose table matches a 16 MiB part there is nothing to assemble. So emit cv6xx lite only, and drop the dv500 loop, which has no lite variant to fall back on. Both comments carry the partition numbers so restoring a loop is a two-line change. #2447 fits both cv6xx lite slots exactly (2048 in 2048, 5088 in 5120) and that image assembles and boots today. The .tgz for both variants still publishes from the build job, so sysupgrade is unaffected -- it splits the blob at the FIT boundary and writes each half to its own partition, and refuses an oversized half rather than mis-landing it.
image is a fixed tag and Upload only adds or replaces the files a run produced, so dropping a target from the assembly loops leaves whatever it published last sitting on the release for good. Six such assets were still downloadable, all dated 2026-09-18: the four hi3516cv610/cv608 ultimate images and both hi3519dv500 ones. They are not merely stale. They were assembled before #2448 taught create_hisi to split the blob at the FIT boundary and measure each half against the partition it goes into, so they carry the layout that PR replaced -- a FIT running past the end of the kernel slot and a rootfs written over rootfs_data. Withdrawing the images from the loops without withdrawing them from the release would have left exactly the files this change exists to stop shipping. Delete them by name before Upload, tolerating an asset that is already absent so the step is idempotent and never fails a run. Drop a name from the list when its loop comes back (#2460).
Problem
create_hisilaid u-boot at 0 and the combinedfirmware.binverbatim at the firmware offset, on a canvas hardcoded to 16384 KiB. Two things were wrong with that, one new and one old.The new one: a lite variant is flashed to an 8 MiB part, and an image whose tail runs past the end of the chip cannot be written to one. The part size is now the sixth argument, defaulting to 16384 so every existing caller keeps its behaviour, and the cv6xx loop emits both variants at the size each is built for.
The old one: the boot images this job pairs with the firmware carry their partition table in a compiled-in env, and nothing rewrites it later. Decompressed from
boot-hi3516cv610-*-nor.bin(a HiSilicon boot table with a gzip'd u-boot behind it):So u-boot reads the kernel as one fixed slot and mounts root at a fixed offset behind it.
firmware.binis the FIT with the squashfs packed at the next 64 KiB boundary (boardpost-image.sh), which only agrees with that table when the FIT happens to end exactly at the slot. On every published cv6xx and dv500 image it does not:root=/dev/mtdblockNpoints 704 KiB before the squashfs on cv6xx ultimate and 512 KiB past it on dv500, and the cv6xx FIT is larger than its 2048 KiB slot, sosf readhandsbootma truncated image.sysupgradealready gets this right (do_update_firmwaresplits the blob at the FIT boundary and writes each half to its partition); a fresh whole-chip flash of the release.bindid not.This does what sysupgrade does: split at the FIT boundary (the FIT's total size is the big-endian word at byte 4; the squashfs length is
bytes_usedin its superblock), put the FIT at the firmware offset and the rootfs at the rootfs partition, and measure each piece against the partition it goes into — u-boot against the boot partition (the env sits behind it, so the firmware offset is the wrong bound), the FIT against the kernel slot, the rootfs against the rootfs slot, the firmware partition against the part. Each failure prints a::error::and fails the step rather than uploading something that boots a truncated kernel or overwritesrootfs_data. The exact-size check on the assembled canvas stays as the backstop, and the scratch variables arelocalnow.The layout table is a
caseon$soc; adding a family means adding its u-boot's env line there. Consequence worth stating: with this in place, cv6xx ultimate on master fails the FIT check (2709 KiB in 2048) and hi3519dv500 ultimate fails the rootfs check (8832 KiB in 8192). Both are the same table disagreement the published images already carry, now reported instead of published. #2447 fits both cv6xx slots.A refusal does not take the other SoCs' images with it. The step runs under Actions' default
bash -e, and the Sigmastar, Allwinner and Ingenic images are assembled before the HiSilicon loops, so a barereturn 1there would have ended the step with their.binfiles sitting intarget/and the upload never reached. The HiSilicon calls collect their failures (|| hisi_failed=1), the step exits 1 after the last one so the run is red, andUploadruns onalways()so what did assemble ships. Exercised in the harness: a refused ultimate followed by a good lite leaves the lite.binintarget/and the step red.Hardware tested on
hi3516cv6xx (chip reports
hi3516cv613, DDR3 128 MB), SIMICAM A314D / Vatilon H80, 8 MiB NOR.This produces the bytes a user writes to a chip, so it was tested as that: the function from this branch, run against the #2447 lite build and the published
boot-hi3516cv610-20s-nor.bin, producedopenipc-hi3516cv610-ddr3-128m-nor-lite.bin, and that image was written to the camera's NOR in place (u-boot, erased env, FIT, rootfs, at the offsets above). It booted on the OpenIPC u-boot with its default env, the kernel reported the table from that env,/initfound and mountedrootfs_data, and after the usualfw_setenv osmem 48Mit streams. u-boot, FIT and squashfs read back byte-identical from their partitions. One board addition:rootfs_datawas pre-staged with the device's overlay (an out-of-tree sensor driver, since this sensor has no driver in the tree) instead of left for/initto format; the rest of the image is the assembled one.Evidence
On the camera, running that image:
Scope
general/overlay/or a shared load scriptLD_PRELOAD, no unbuildable binaries