Boundary layer generation intermittently fails or produces invalid meshes

[zasexton]

Description

Boundary layer generation occasionally fails across multiple retries or produces invalid geometry/meshes (non-manifold surfaces, inverted elements, or negative Jacobians). This happens both on simple shapes (e.g., cylinder) and on procedurally generated trees, making 3D CFD setup unreliable when boundary_layer=True.

Reproduction

Build a simple cylinder, tetrahedralize, extract the wall, then generate layers:

        import pyvista as pv
        import tetgen
        from svv.simulation.utils.extract_faces import extract_faces
        from svv.simulation.utils.boundary_layer import BoundaryLayer

        cyl = pv.Cylinder().triangulate()
        t = tetgen.TetGen(cyl)
        t.tetrahedralize()
        surf = t.grid.extract_surface()
        _, walls, _, _ = extract_faces(surf, t.grid)
        wall = walls[0]

        bl = BoundaryLayer(wall, layer_thickness=0.2*surf.hsize if hasattr(surf, 'hsize') else 0.1, remesh_vol=False)
        combined, interior, layers = bl.generate()  # often raises or yields non-manifold

We get intermittent exceptions such as “Merged Outer Surface is not manifold”, mapping 1:1 failures, or negative Jacobian warnings.

Expected behavior

• Boundary layer generation succeeds reliably for simple and moderately complex walls.
• Resulting combined mesh is manifold with no inverted surface triangles and no negative Jacobians.
• When failure occurs, it degrades gracefully or auto-recovers (e.g., auto-orient normals or invert warp direction)
  rather than hard failing.

Additional context

Potential causes:

• Normal flip detection bug: check_tangle() seems ineffective because the per-cell normal comparison uses a norm that
  is never negative:
  • svv/simulation/utils/boundary_layer.py:281
    • current: np.linalg.norm(base_normals * warped_normals, axis=1) < 0.0 is always False
    • effect: inverted faces go undetected; later steps can produce invalid layers
• Outer surface merge is fragile on caps:
  • Cap construction + pv.merge(...).clean(...) can create non-manifold merged outer surfaces, triggering a hard
    failure
  • svv/simulation/utils/boundary_layer.py:549 and :552
• Mapping constraints (inner->outer, inner->mesh) are strict 1:1 and conforming; small numerical misalignments or point
  duplication along splits can violate these constraints and abort
  • svv/simulation/utils/boundary_layer.py:660, :666, :669
• Normal orientation and warp direction are brittle:
  • Reliance on surface normals plus negate_warp_vectors can send layers inward on odd patches; there’s no auto-
    detection/fallback to flip direction

Code of Conduct

• I agree to follow this project's Code of Conduct and Contributing Guidelines