Refuse consolidating a voxel or mesh layer as unsupported (#659)

[leonardoaraujosantos]

Problem

Consolidating a voxel or a mesh layer failed with CLAY_ERROR_INVALID_ARGUMENT and the detail meant for an empty SDF layer:

clay_layer_consolidate -> CLAY_ERROR_INVALID_ARGUMENT
"nothing to consolidate: the layer is empty, unbounded, or the region contains no surface"

ClaySpaceDesktop showed that detail on layer/optimize over a grid layer, and artists read it as "already optimised". The real answer is "this only applies to SDF layers". A host could not tell "wrong kind of layer" from "nothing there yet" without first asking clay_document_layer_info.

Root cause

clay_layer_consolidation_cost, clay_layer_consolidate_cancellable (and clay_layer_consolidate through it) and clay_layer_consolidate_region checked only for a missing or a protected layer, then went to the bake. The bake returns false for a layer with no SDF content (src/scene/consolidate.cpp), and that false maps to "nothing to consolidate" / "nothing to merge". clay_layer_plan_region_merge had no check at all: it succeeded with a plan that absorbs nothing, for a merge the commit would then refuse.

Fix

find_consolidatable in bindings/c/clay_c.cpp now resolves the layer for all five calls, in this order:

1. CLAY_ERROR_NOT_FOUND when the layer is missing.
2. CLAY_ERROR_UNSUPPORTED for a voxel or a mesh layer. clay_last_error reads "consolidation applies to SDF layers: layer N is a voxel layer" (or "a mesh layer"), the same wording clay_document_instance_layer uses.
3. For the calls that edit, the existing protected-layer CLAY_ERROR_INVALID_ARGUMENT.

All of this runs before any params are read and before anything is sampled. Representation goes before protection on purpose: a locked grid would otherwise answer "locked" and send the artist to unlock a layer that can never be consolidated.

An empty SDF layer still answers CLAY_ERROR_INVALID_ARGUMENT "nothing to consolidate", so the two cases stay distinct. clay_layer_plan_region_merge still plans zero absorbed roots for an empty SDF layer.

Deliberately not changed: clay_layer_consolidation_advice and clay_layer_consolidation_state.

• The triage plan put the advice behind the refusal too. Its header has said since 0.86.0 that a non-SDF layer "is NOT an error — it succeeds, advises nothing and zeroes", following clay_layer_warp_cost_get's rule for a stack of mixed kinds, and an existing test asserts that.
• "Should I bake this?" has an answer for a grid. "Bake this" does not.
• The new test pins the advice's answer for a voxel and a mesh layer.

Folding the three checks into one helper (plus a small read_optional_merge) also brings clay_layer_consolidate_region from cognitive complexity 19 to 15. _cancellable goes from 14 to 13. Measured with clang-tidy readability-function-cognitive-complexity.

Docs and spec

• The clay.h comments on cost, consolidate, _cancellable, plan_region_merge and consolidate_region list the new error. The advice comment says why it is the exception.
• docs/05 gets a paragraph in the consolidation-advice section.
• New OpenSpec change refuse-consolidating-a-voxel-or-mesh-layer modifies "Consolidation across the ABI". It repeats the two existing scenarios and adds "A voxel or mesh layer is refused as unsupported" and "An empty SDF layer still has nothing to consolidate".

Regression test

In tests/unit/test_c_consolidate.cpp:

• "consolidating a voxel or mesh layer is refused as unsupported, by name" runs one subcase on a voxel layer and one on a mesh layer. Each one calls cost, consolidate, _cancellable (with a token), plan_region_merge and consolidate_region, and asserts:
  • every call returns CLAY_ERROR_UNSUPPORTED;
  • clay_last_error() names "voxel" or "mesh";
  • the undo depth is unchanged;
  • a locked layer is still UNSUPPORTED;
  • the advice still answers OK, not advised, with zeroed params;
  • _state answers 0.
• "an empty SDF layer still answers nothing to consolidate, not unsupported" is the control.

Proof it failed before: I ran the same test file against unfixed origin/main (02d9672). The new case failed 24 assertions (12 per representation: 1 == 7 for cost, consolidate, _cancellable and consolidate_region, 0 == 7 for plan_region_merge, and every last_error_names check). The control and advice checks passed there, as intended. With the fix: 19/19 cases, 622/622 assertions.

Verification

All on macOS, cpu-only preset with CLAY_BUILD_TESTS=ON, exit codes checked directly:

• Full build passes.
• ctest: 10/10, including all four unit shards.
• check_c_abi.py against the freshly built libclay_shared.dylib: OK.
• check_layering, check_kernel_dialect, check_licenses, check_doc_latency and check_test_shards --binary: OK (3029 cases partitioned).
• openspec@1.12.0 validate --all --strict: 76/76.
• release_check.py --skip-slow: these rows PASS: version, configure, build, tests (11/11), parity, layering, dialect, licenses, task-symbols, bindings (pyclay imported), kernels, abi, openspec.
  • It still fails on device and four hardware/* waivers. Those are release-time gates and are already stale on main: the device gate last ran at 7f6cf38, and the hardware waivers trip on include/clay/eval/bake_volume.h, which this PR does not touch.
• GCC 12 -Werror (the manylinux job) cannot run on macOS. The added code is plain if / std::string concatenation with the assignment-in-condition parenthesised.

ABI

No entry-point, struct, enumerator, format or version change. The ABI stays at 0.123.0. This is a behaviour change on an error path only:

• A voxel or mesh layer now gets CLAY_ERROR_UNSUPPORTED where it used to get CLAY_ERROR_INVALID_ARGUMENT.
• clay_layer_plan_region_merge on such a layer now fails instead of returning an empty plan.

Closes #659