cvcGL tests: GL tests hold one resource lock; shadow_casters D prints its macOS numbers - #529
Merged
Merged
Conversation
On the macOS runners (Apple Software Renderer, GL 4.1) the first attempt of cvcgl_shadow_casters failed in every Debug package job seen (#521 x2, #522, #523), always the same way: luma 0.0 in BOTH halves of all three captures (casting, non-casting, absent), so "the ground is lit" and "a casting slab darkens the ground" failed together while every bake-decision check passed; ctest's retry then passed. Not a threshold problem: no image of the ground was read at all. The ground was a 120x120 box: two top triangles viewed through a 16-unit window, so they were clipped far outside the frustum, and the shadow bake's light frustum is fitted to the scene bounds, so the ground's edges lay on it too. Apple's software renderer intermittently drops triangles like that. With a black background, a ground lost from the frame and a ground lost from the shadow maps (which VTK's exponential maps read as total shadow -- confirmed here: a map without the ground blacks out the whole ground) both read 0.0. - The ground is now 2-unit tiles (tiledGround), so no triangle reaches far outside either frustum. - The background is blue (red still 0, so the map clear is unchanged): every capture must show the ground in all of its pixels, and a capture that lost it says so instead of passing for shadow. On Apple only, such a capture is redrawn (same maps, no bake) up to 3 times, each redraw logged; elsewhere the first draw must be whole. - Each capture follows a settle (frames until no bake), and the frame size must be 96x96 before any pixel is indexed. - Thresholds come from the same run: casting must fall under half of the same frame's lit left half; not casting must stay within 10% of it. - Strict where rendering is deterministic: non-casting is byte-identical to absent over the whole frame, and casting leaves the left half byte-identical. On Apple only, the same claims are held to the 1-luma bound the absent/non-casting comparison always met there. - D prints the GL renderer string, for the next macOS log. Linux: 30/30 NVIDIA and 30/30 llvmpipe (plus 24 concurrent llvmpipe and ctest --repeat until-fail:30), all byte-exact. Breaking the caster filter or the caster re-bake fails D; a ground smaller than the view fails the coverage check.
…ext) Every cvcGL test that renders through a GL context now takes the CTest resource lock cvcgl_gl_context, so ctest never runs two of them at once; the 23 headless tests (listed by name) keep running in parallel. The lock is assigned by exclusion -- every test in the directory minus the headless list -- so a GL test added later is locked unless someone lists it as headless. Why: on the macOS runners every failed attempt of cvcgl_shadow_casters (4 of the 12 attempts across the 8 package-macos jobs of #521-#523) ran alongside cvcgl_shadow_caster_growth and other GL tests under ctest --parallel; no passing attempt overlapped growth, and each retry passed. That is a correlation, not a reproduction -- no Mac reproduced the flake. On every platform, on purpose: the cost is wall time only. On macOS Debug the GL tests' own times summed to ~370 s (cvcgl_volren_node 230 s of it) where they overlapped into ~250 s, so that ctest step can grow by up to ~2 minutes of ~9.
… redraws Corrects ae82da2, which stated a cause it had not shown. Diagnosis, restated. On the macOS runners the first ctest attempt of cvcgl_shadow_casters failed in every Debug job seen (#521 x2, #522, #523), luma 0.0 everywhere in D; each retry passed; Release never failed. The leading explanation is contention: all 4 failing attempts ran while cvcgl_shadow_caster_growth was rendering, and none of the 8 passing attempts did (retries and Release runs overlapped other GL tests, but not growth). The previous commit's cause -- Apple's GL dropping large triangles clipped far outside the frustum -- is a hypothesis nothing has confirmed; it does not explain why a retry of the same binary seconds later always passes. A shadow map sampled as empty fits "0.0 everywhere" as well. The GL tests now hold one CTest resource lock (previous commit); whether that ends it is for the next macOS Debug runs to show. - The tiled ground stays, described as a guard against that hypothesis, not as the fix. - The Apple-only redraw loop (up to 3 redraws of a frame missing ground) is gone. Its cap was a guess, and ctest hid the redraw lines of any passing run, so it could never be measured. Every capture must now show the whole ground on its first draw, on every renderer; a frame that does not fails, and ctest's retry runs the test again. The blue background still tells the two failure modes apart: ground missing from the frame counts background pixels, ground drawn black (shadow map read as empty) counts none and reads luma 0. - D prints the renderer string, each capture's background-pixel count, the luma values and, new, the differing-byte count and largest step of non-caster vs absent (whole frame) and caster vs non-caster (left half), pass or fail. The next commit prints this section from passing macOS runs. - The Apple relaxation is keyed on the renderer string containing "Apple" inside the __APPLE__ guard, not on the platform alone. There it is master's check, unchanged: the footprint's mean luma within 1 of the slab-absent frame's. That tolerance is carried over, not measured -- no passing run ever printed the values, and the failing runs read 0.0 vs 0.0. The previous commit's new left-half claim (|lLeft - lLeftNot| < 1) is not made on Apple; its numbers are printed for a later bound. - Everywhere else, the two claims are byte-exact, now as separate checks: non-caster frame == absent frame, and casting leaves the left half as it was. Linux, CVC_REQUIRE_RENDER=1: 30/30 NVIDIA GTX 1650 and 30/30 llvmpipe (LLVM 20.1.2), 0 differing bytes in every run, identical luma lines. The Apple branch, forced on with a renderer-string key that matches llvmpipe and not NVIDIA, passes on llvmpipe while NVIDIA stays byte-exact. Mutants fail D: capturing "not casting" with the slab still casting (non-caster footprint 0.0, 11882 bytes differ); a 10x10 ground (5616 background pixels in each capture); the light moved for the casting capture only (5184 bytes differ in the left half).
…runs ctest --output-on-failure keeps a passing test's output out of the job log. A flake that passes on retry then leaves no numbers behind, and a tolerance that always passes is never measured. For cvcgl_shadow_casters that hid the renderer string, the background-pixel counts, the luma values and the differing-byte counts on macOS, where D keeps master's carried-over 1-luma check and does not yet judge the left half. ctest's LastTest.log keeps every attempt's output. A new package-macos step, run always(), prints D from each cvcgl_shadow_casters attempt in it, with its verdict. A Release push runs a second ctest after the first, which replaces LastTest.log, so that one case prints nothing; PRs and Debug pushes run a single ctest. The awk program gives the same output under gawk, mawk, nawk and busybox awk on a LastTest.log with three attempts.
…printed The failing runs printed the three footprints and the casting frame's left half, not both halves of every capture. Also names cvcgl_stage_caster_truth next to growth, and notes that in the Debug jobs the first attempt and the overlap always coincided, so a first-run effect is not ruled out.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
cvcgl_shadow_casters(#521) failed its first ctest attempt in every macOS Debug job seen so far (#521 x2, #522, #523), always with luma 0.0 in every value it printed, and passed each retry.Diagnosis (leading explanation, not confirmed)
Contention. All 4 failing first attempts ran while
cvcgl_shadow_caster_growthandcvcgl_stage_caster_truth(other shadow-baking tests) were rendering. None of the 8 passing attempts overlapped growth. In the Debug jobs, "first attempt" and "overlapped growth" always coincided, so a first-run effect is not ruled out. No Mac has reproduced it locally. (Commit ae82da2's message names a triangle-drop cause; 3256619 corrects that. The tiled ground stays only as a guard against that hypothesis.)Changes
RESOURCE_LOCK cvcgl_gl_context, all platforms). It is assigned by exclusion: every test insrc/cvcGLexcept a named list of 23 headless tests, so a new GL test is locked by default. 23 tests are locked and 23 are not. It costs wall time only; the macOS Debug ctest step may grow by about 2 minutes. cvcGL: fast-draw low-memory mapper (~2/3 fewer GL calls per frame on WebGL2) + reship cvc.16 #522 carries the identical hunk (same patch-id), so either merge order is clean.package-macosstep (if: always()) prints test D's section from everycvcgl_shadow_castersattempt inLastTest.log, pass or fail. Gap: on a Release push a second ctest overwritesLastTest.log.Testing (Linux,
CVC_REQUIRE_RENDER=1)ctest -R cvcgl_ -j8passes 46/46, and no two locked tests overlap.The macOS result can only come from this PR's CI. Read the first attempt's D section in both macOS jobs, not just the job status.