[Televiz] ProjectionLayer + in-loop frame protocol refactor#544
[Televiz] ProjectionLayer + in-loop frame protocol refactor#544farbod-nv wants to merge 1 commit into
Conversation
📝 WalkthroughWalkthroughThis PR introduces Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes The PR introduces significant new functionality (ProjectionLayer implementation, frame lifecycle refactor, shader programs, Python bindings) across heterogeneous files with dense logic in submission, rendering, and frame synchronization paths. The frame lifecycle refactor affects the compositor core and session state management, requiring careful review of exception handling and RAII patterns. Multiple independent subsystems (Vulkan resource management, CUDA interop, descriptor set binding, pipeline selection) must be verified separately for correctness. 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Infer (1.2.0)src/viz/core/cpp/device_image.cppsrc/viz/core/cpp/device_image.cpp:4:10: fatal error: 'viz/core/device_image.hpp' file not found ... [truncated 1119 characters] ... l/lib/clang/18/include" src/viz/layers/cpp/projection_layer.cppsrc/viz/layers/cpp/projection_layer.cpp:4:10: fatal error: 'viz/core/render_target.hpp' file not found ... [truncated 1138 characters] ... clang/18/include" src/viz/layers_tests/cpp/test_projection_layer.cppsrc/viz/layers_tests/cpp/test_projection_layer.cpp:9:10: fatal error: 'test_helpers.hpp' file not found ... [truncated 1139 characters] ... e"
Comment |
Adds ProjectionLayer for full-view RGBD content (gsplat, nvblox,
neural reconstruction) and refactors VizSession's frame loop so the
poses fed to renderers match the poses submitted to OpenXR.
## ProjectionLayer
In-loop contract:
info = session.begin_frame() # xrWaitFrame + Begin + LocateViews
color, depth = renderer.render(info.views) # render against THIS frame's views
layer.submit(color, depth) # publish for this frame
session.end_frame() # composite + xrEndFrame
The runtime / CloudXR paces the app via xrWaitFrame; if the renderer
takes 30 ms, the app runs at ~30 fps and the runtime compositor
reprojects the last submitted frame at display rate.
Storage: per-slot (color, depth) DeviceImages (7-slot mailbox per
QuadLayer's pattern). Fragment shader samples color + depth and writes
gl_FragDepth so QuadLayer / future OverlayLayer Z-composite correctly
in the shared RT. Stereo: paired (left, right) slot storage; submit()
ships both eyes on one CUDA stream.
In kXr, a visible ProjectionLayer that doesn't submit for the current
frame is skipped at record() time so stale RGBD never gets composited
under a new projection-layer pose. The freshness gate is off in
kWindow / kOffscreen where no XR pose mismatch is possible.
Single XrCompositionLayerProjection covers QuadLayers + ProjectionLayer
together — CloudXR fast-paths when only ProjectionLayer is present,
and squashes per-layer-timewarped when mixed (Monado's compute-shader
layer accumulator handles this automatically).
## VizSession frame-loop refactor
``VizSession::begin_frame`` previously returned placeholder identity
views; the real per-eye XR poses were only acquired later inside
``VizCompositor::render`` via ``backend_->begin_frame``. That meant
renderers calling ``submit`` against the returned FrameInfo were
rendering against the wrong pose.
Moves backend frame acquisition into ``VizSession::begin_frame``:
acquired Frame is stored in ``current_backend_frame_`` and consumed
by ``end_frame``. ``FrameInfo.views`` now carries the real xrLocateViews
output, so the in-loop pattern above is pose-correct by construction.
VizCompositor::render takes the acquired Frame as a parameter; the
existing FrameGuard still owns end_frame/abort_frame protocol balance.
Adds Frame::predicted_display_time_ns so XR's
``last_frame_state_.predictedDisplayTime`` flows through to
``FrameInfo.predicted_display_time``.
## LayerBase::on_frame_begin
New virtual called from VizSession::begin_frame on every registered
layer (default no-op). ProjectionLayer uses it to clear its
submitted-this-frame freshness flag.
## Minor changes
- ``DeviceImage`` allows PixelFormat::kD32F (was rejected as
"reserved for ProjectionLayer"). kD32F + mip_levels > 1 rejected
explicitly.
- Python bindings: expose ``ViewInfo`` and ``FrameInfo.views``.
No ``ProjectionViewSnapshot`` / ``predicted_views`` / ``get_last_frame_info``
— the renderer reads ``info.views`` returned by ``begin_frame``.
## Tests
- 13 C++ Catch2 tests (4 unit + 9 GPU): config validation, slot/view
allocation, idempotent destroy, submit shape/format/dimension
validation, mono+depth mailbox advance, stereo paired submit,
no-depth path, on_frame_begin/submitted-this-frame toggling.
- 8 Python pytest tests covering the same surface plus the in-loop
submit pattern via begin_frame/end_frame.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
farbod-nv
force-pushed
the
fm/projectionlayer
branch
from
af669a5 to
e1c8895
Compare
farbod-nv
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@src/viz/layers_tests/cpp/test_projection_layer.cpp`:
- Around line 86-95: The GPU skip gate must also catch initialization failures:
wrap the Vulkan setup calls so that exceptions from VkContext::init({}) and
RenderTarget::create(...) are converted into SKIP(...) instead of letting them
throw; keep the initial is_gpu_available() check, then try/catch around ctx.init
and creation of target (referencing VkContext::init, RenderTarget::create,
RenderTarget::Config and Resolution) and call SKIP("...") with a brief message
when an exception is caught so the test suite skips cleanly on runners without
working Vulkan.
In `@src/viz/python/layers_bindings.cpp`:
- Around line 184-234: The binding for ProjectionLayer.submit in
layers_bindings.cpp currently throws std::runtime_error only for the left_color
nil case but lets std::invalid_argument from viz::ProjectionLayer::submit
propagate (which becomes Python ValueError), causing test mismatch; wrap the
call to self.submit(...) in a try/catch that catches std::invalid_argument
(thrown by the C++ submit/validation in src/viz/layers/cpp/projection_layer.cpp)
and rethrow as std::runtime_error so pybind11 converts it to Python
RuntimeError; update the catch to include the original exception message when
constructing the std::runtime_error to preserve error details.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 9d2a40b9-036d-453c-95d3-6a3b59e9b9ba
📒 Files selected for processing (22)
src/viz/core/cpp/device_image.cppsrc/viz/layers/cpp/CMakeLists.txtsrc/viz/layers/cpp/inc/viz/layers/projection_layer.hppsrc/viz/layers/cpp/projection_layer.cppsrc/viz/layers_tests/cpp/CMakeLists.txtsrc/viz/layers_tests/cpp/test_projection_layer.cppsrc/viz/python/core_bindings.cppsrc/viz/python/layers_bindings.cppsrc/viz/python/session_bindings.cppsrc/viz/python/viz_init.pysrc/viz/python_tests/test_projection_layer.pysrc/viz/session/cpp/inc/viz/session/display_backend.hppsrc/viz/session/cpp/inc/viz/session/layer_base.hppsrc/viz/session/cpp/inc/viz/session/viz_compositor.hppsrc/viz/session/cpp/inc/viz/session/viz_session.hppsrc/viz/session/cpp/viz_compositor.cppsrc/viz/session/cpp/viz_session.cppsrc/viz/session/cpp/xr_backend.cppsrc/viz/shaders/cpp/CMakeLists.txtsrc/viz/shaders/cpp/projection_layer.fragsrc/viz/shaders/cpp/projection_layer.vertsrc/viz/shaders/cpp/projection_layer_no_depth.frag
| TEST_CASE("ProjectionLayer mono+depth creates valid handles for every slot+view", "[gpu][projection_layer]") | ||
| { | ||
| if (!is_gpu_available()) | ||
| { | ||
| SKIP("No Vulkan-capable GPU available"); | ||
| } | ||
| VkContext ctx; | ||
| ctx.init({}); | ||
| auto target = RenderTarget::create(ctx, RenderTarget::Config{ Resolution{ 64, 64 } }); | ||
|
|
There was a problem hiding this comment.
GPU skip gate is incomplete; Vulkan init exceptions currently fail tests.
is_gpu_available() alone isn’t preventing hard failures when ctx.init({}) throws (seen in CI). Convert init/setup failures into SKIP(...) so GPU tests skip cleanly on unsupported runners instead of failing the suite.
Proposed fix (apply pattern to all [gpu] tests)
+static bool init_vk_or_skip(VkContext& ctx)
+{
+ try
+ {
+ ctx.init({});
+ return true;
+ }
+ catch (const std::exception&)
+ {
+ return false;
+ }
+}
+
TEST_CASE("ProjectionLayer mono+depth creates valid handles for every slot+view", "[gpu][projection_layer]")
{
if (!is_gpu_available())
{
SKIP("No Vulkan-capable GPU available");
}
VkContext ctx;
- ctx.init({});
+ if (!init_vk_or_skip(ctx))
+ {
+ SKIP("Vulkan initialization failed on this runner");
+ }
auto target = RenderTarget::create(ctx, RenderTarget::Config{ Resolution{ 64, 64 } });🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/viz/layers_tests/cpp/test_projection_layer.cpp` around lines 86 - 95, The
GPU skip gate must also catch initialization failures: wrap the Vulkan setup
calls so that exceptions from VkContext::init({}) and RenderTarget::create(...)
are converted into SKIP(...) instead of letting them throw; keep the initial
is_gpu_available() check, then try/catch around ctx.init and creation of target
(referencing VkContext::init, RenderTarget::create, RenderTarget::Config and
Resolution) and call SKIP("...") with a brief message when an exception is
caught so the test suite skips cleanly on runners without working Vulkan.
Source: Pipeline failures
| .def( | ||
| "submit", | ||
| [](viz::ProjectionLayer& self, py::object left_color, py::object left_depth, py::object right_color, | ||
| py::object right_depth, uintptr_t stream) | ||
| { | ||
| auto to_buf = [&self](py::object obj, viz::PixelFormat fmt, const char* label) -> viz::VizBuffer | ||
| { | ||
| if (py::isinstance<viz::VizBuffer>(obj)) | ||
| { | ||
| return obj.cast<viz::VizBuffer>(); | ||
| } | ||
| return cuda_array_to_viz_buffer(obj, fmt, self.view_resolution(), label); | ||
| }; | ||
|
|
||
| // Materialize each buffer (or std::nullopt). View slots | ||
| // that aren't provided pass nullptr through to submit. | ||
| std::optional<viz::VizBuffer> lc; | ||
| std::optional<viz::VizBuffer> ld; | ||
| std::optional<viz::VizBuffer> rc; | ||
| std::optional<viz::VizBuffer> rd; | ||
| if (!left_color.is_none()) | ||
| { | ||
| lc = to_buf(left_color, self.color_format(), "ProjectionLayer.submit(left_color)"); | ||
| } | ||
| else | ||
| { | ||
| throw std::runtime_error("ProjectionLayer.submit: left_color is required"); | ||
| } | ||
| if (!left_depth.is_none()) | ||
| { | ||
| ld = to_buf(left_depth, viz::PixelFormat::kD32F, "ProjectionLayer.submit(left_depth)"); | ||
| } | ||
| if (!right_color.is_none()) | ||
| { | ||
| rc = to_buf(right_color, self.color_format(), "ProjectionLayer.submit(right_color)"); | ||
| } | ||
| if (!right_depth.is_none()) | ||
| { | ||
| rd = to_buf(right_depth, viz::PixelFormat::kD32F, "ProjectionLayer.submit(right_depth)"); | ||
| } | ||
|
|
||
| py::gil_scoped_release release; | ||
| self.submit(*lc, ld.has_value() ? &*ld : nullptr, rc.has_value() ? &*rc : nullptr, | ||
| rd.has_value() ? &*rd : nullptr, reinterpret_cast<cudaStream_t>(stream)); | ||
| }, | ||
| "left_color"_a, "left_depth"_a = py::none(), "right_color"_a = py::none(), "right_depth"_a = py::none(), | ||
| "stream"_a = 0, | ||
| "Submit a frame. Each arg is a VizBuffer or any __cuda_array_interface__ object. " | ||
| "Mono: only ``left_color`` (+ ``left_depth`` if depth-enabled). " | ||
| "Stereo: pair with ``right_color`` (+ depths). Buffers must match view_resolution " | ||
| "and the layer's pixel formats. Releases the GIL across the copy + sync.") |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Check for custom pybind11 exception translators in the bindings
rg -n "register_exception" src/viz/python/ -A 5
# Look for exception_translator or translate_exception
rg -n "translate.*exception|exception.*translator" src/viz/python/ -A 5
# Check what validate_submit_buffer actually throws
ast-grep --pattern 'throw $_(validate_submit_buffer'Repository: NVIDIA/IsaacTeleop
Length of output: 44
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# 1) Inspect where submit/validate throws std::invalid_argument in the C++ layer
echo "== validate/submit exception sites =="
rg -n "validate_submit_buffer|invalid_argument" src/viz/layers -S
rg -n "validate_submit_buffer" src/viz -S
# Show the relevant region around the specific line numbers mentioned (496/513/530) if present
for f in $(rg -l "validate_submit_buffer|ProjectionLayer::submit" src/viz | head -n 20); do
echo "---- $f ----"
nl -ba "$f" | sed -n '450,580p' | sed -n '1,200p'
done
# 2) Search the entire python bindings directory for any custom exception translators
echo "== custom exception translator registrations in src/viz/python =="
rg -n "register_exception|translate_exception|exception_translator|PyExc|PyErr_SetString" src/viz/python -S
# 3) Search for pybind11 module initialization / translation elsewhere under src/viz
echo "== custom exception translator registrations in src/viz (broader) =="
rg -n "register_exception|translate_exception|exception_translator" src/viz -S
# 4) Inspect Python test expectations
echo "== test expectations =="
fd -a "test_projection_layer.py" -S . | head -n 20
python - <<'PY'
import pathlib, re
paths=[]
for p in pathlib.Path('.').rglob('test_projection_layer.py'):
paths.append(p)
print("found:", [str(x) for x in paths])
for p in paths:
txt=p.read_text(encoding='utf-8')
# Print around the referenced lines 110-220 if possible
lines=txt.splitlines()
for i in [110,120,130,140,160,170,190,200,210,220]:
if 1<=i<=len(lines):
pass
start=100
end=min(240,len(lines))
print("\n----",p,"lines",start,"-",end,"----")
for idx in range(start-1,end):
if idx<0: continue
print(f"{idx+1:4d}: {lines[idx]}")
PYRepository: NVIDIA/IsaacTeleop
Length of output: 6289
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== validate/submit exception sites (projection_layer) =="
rg -n "ProjectionLayer::validate_submit_buffer|throw std::invalid_argument|invalid_argument" src/viz/layers/cpp/projection_layer.cpp -n -S
echo
echo "== Python bindings: exception translator hooks in src/viz/python =="
rg -n "register_exception|translate_exception|exception_translator|PyErr_SetString" src/viz/python -S || true
echo
echo "== Python bindings: broader pybind11 exception translator hooks under src/viz =="
rg -n "register_exception|translate_exception|exception_translator|PyErr_SetString" src/viz -S || true
echo
echo "== Locate test_projection_layer.py and inspect expected exception types =="
fd -a "test_projection_layer.py" -S . || true
for p in $(fd -a "test_projection_layer.py" -S . 2>/dev/null); do
echo "---- $p ----"
# print a window around the previously referenced lines 100-230
sed -n '90,240p' "$p" | cat
done
echo
echo "== Locate C++ tests (if any) for submit validation =="
fd -a "test_projection_layer.*" -S src/viz | head -n 50 || true
for p in $(fd -a "test_projection_layer.*" -S src/viz 2>/dev/null); do
echo "---- $p ----"
rg -n "invalid_argument|runtime_error|EXPECT_THROW|throw" "$p" -S || true
doneRepository: NVIDIA/IsaacTeleop
Length of output: 2245
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Locate python tests referencing ProjectionLayer.submit =="
rg -n "test_projection_layer\.py|ProjectionLayer\.submit|submit\(left_color" -S . || true
echo
echo "== Search for RuntimeError/TypeError expectations around submit =="
rg -n "pytest\.raises\((RuntimeError|TypeError)|RuntimeError\)|TypeError\)" -S . || true
echo
echo "== Search for any pybind11 exception translator customizations =="
rg -n "register_exception|translate_exception|exception_translator|PyErr_SetString|py::register_exception|pybind11::register_exception|throw_exception|PYBIND11_MODULE" src/viz/python -S || true
rg -n "register_exception|translate_exception|exception_translator|PyErr_SetString|py::register_exception|PYBIND11_MODULE" src/viz -S || true
echo
echo "== Inspect python module init for exception translators (if any) =="
rg -n "PYBIND11_MODULE" -S src/viz/python src/viz || true
echo
echo "== Locate validate_submit_buffer call sites in bindings (to connect to exceptions) =="
rg -n "ProjectionLayer.*submit|def\(\s*\"submit\"|layers_bindings\.cpp" -S src/viz/python || trueRepository: NVIDIA/IsaacTeleop
Length of output: 10212
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== layers_bindings.cpp: submit binding (look for try/catch/translation) =="
p="src/viz/python/layers_bindings.cpp"
if [ -f "$p" ]; then
sed -n '150,270p' "$p" | cat
else
echo "missing $p"
fi
echo
echo "== Search src/viz/python for any invalid_argument handling =="
rg -n "invalid_argument" src/viz/python -S || true
echo
echo "== Search for pybind11 translator APIs specifically =="
rg -n "register_exception_translator|register_exception|translate_exception|exception_translator" src/viz/python -S || true
rg -n "py::.*register_exception|py::.*translate" src/viz/python -S || true
echo
echo "== Inspect top-level binding init for potential exception translation =="
for f in src/viz/python/viz_bindings.cpp src/viz/python/session_bindings.cpp src/viz/python/bindings_helpers.hpp; do
echo "---- $f ----"
if [ -f "$f" ]; then
sed -n '1,120p' "$f" | cat
sed -n '120,240p' "$f" | cat
else
echo "missing $f"
fi
doneRepository: NVIDIA/IsaacTeleop
Length of output: 25270
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Show test_projection_layer.py around lines 110-220 =="
p="src/viz/python_tests/test_projection_layer.py"
if [ -f "$p" ]; then
sed -n '100,240p' "$p" | cat
else
echo "missing $p"
ls -R src/viz/python_tests | head -n 200
fi
echo
echo "== Search for any py::register_exception_translator usage (anywhere) =="
rg -n "py::register_exception_translator|register_exception_translator" src/viz -S || true
echo
echo "== Search for custom exception translator lambdas by type (broader) =="
rg -n "translate_exception|exception_translator|register_exception" src/viz/python -S || true
echo
echo "== Search for std::invalid_argument handling in bindings =="
rg -n "catch \\(const std::invalid_argument|invalid_argument\\)" src/viz/python -S || trueRepository: NVIDIA/IsaacTeleop
Length of output: 4991
🌐 Web query:
pybind11 default exception translation std::invalid_argument maps to which Python exception type
💡 Result:
In pybind11, the C++ standard exception std::invalid_argument is automatically translated to the Python ValueError exception type [1][2]. This translation is handled by pybind11's default exception converter, which maps several common C++ standard exceptions to their corresponding built-in Python exceptions [1][3]. While these are default mappings, developers can override this behavior or extend translation for other types by using py::register_exception_translator [1][4].
Citations:
- 1: https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html
- 2: https://github.com/pybind/pybind11/blob/master/docs/advanced/exceptions.rst
- 3: https://git.acem.ece.illinois.edu/lib/pybind11-2.10.4/src/commit/fba01da1c3671f6f0ee3adf0eaaa3b8d98e28e08/docs/advanced/exceptions.rst
- 4: http://pybind11-jagerman.readthedocs.io/en/latest/advanced/exceptions.html
🏁 Script executed:
#!/bin/bash
set -euo pipefail
f="src/viz/layers/cpp/projection_layer.cpp"
echo "== projection_layer.cpp: locate submit + catches =="
rg -n "ProjectionLayer::submit|try \\{|catch \\(" "$f" -S
echo
echo "== projection_layer.cpp: show submit region =="
# show a window around the known validation call sites (~480-560) plus submit definition
sed -n '430,560p' "$f" | cat
echo
echo "== projection_layer.cpp: show any catch blocks near submit/validate =="
rg -n "catch \\(" "$f" -SRepository: NVIDIA/IsaacTeleop
Length of output: 4898
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "py::register_exception_translator|register_exception_translator|exception_translator|translate_exception|PYBIND11_MODULE" . -S || trueRepository: NVIDIA/IsaacTeleop
Length of output: 1412
🌐 Web query:
pybind11 std::invalid_argument maps to RuntimeError vs ValueError
💡 Result:
pybind11 maps C++ std::invalid_argument to the Python ValueError by default (not RuntimeError). [1] Note: pybind11’s default exception translation table explicitly lists “std::invalid_argument -> ValueError”. [1][2] If you register a custom exception translator, you can override this mapping for exceptions you catch and rethrow, and which translator “wins” depends on module import/registration order. [1][2]
Citations:
- 1: https://pybind11.readthedocs.io/en/stable/advanced/exceptions.html
- 2: https://github.com/pybind/pybind11/blob/master/docs/advanced/exceptions.rst
Fix exception type mismatch for ProjectionLayer.submit validation errors
src/viz/layers/cpp/projection_layer.cpp throws std::invalid_argument for submit/validation failures (e.g., missing depth, resolution mismatch, missing right buffers). The Python binding in src/viz/python/layers_bindings.cpp does not catch these (it only throws std::runtime_error for the binding-side left_color is required case), so pybind11’s default translation maps std::invalid_argument → Python ValueError, not RuntimeError.
src/viz/python_tests/test_projection_layer.py expects RuntimeError for these failures (lines 126, 143, 173, 205), so the exception type expectations/API contract are inconsistent.
Fix by either (a) updating the tests to expect ValueError, or (b) catching std::invalid_argument in the submit binding wrapper and rethrowing std::runtime_error (or registering an explicit pybind11 translator).
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@src/viz/python/layers_bindings.cpp` around lines 184 - 234, The binding for
ProjectionLayer.submit in layers_bindings.cpp currently throws
std::runtime_error only for the left_color nil case but lets
std::invalid_argument from viz::ProjectionLayer::submit propagate (which becomes
Python ValueError), causing test mismatch; wrap the call to self.submit(...) in
a try/catch that catches std::invalid_argument (thrown by the C++
submit/validation in src/viz/layers/cpp/projection_layer.cpp) and rethrow as
std::runtime_error so pybind11 converts it to Python RuntimeError; update the
catch to include the original exception message when constructing the
std::runtime_error to preserve error details.
2 participants
Adds ProjectionLayer for full-view RGBD content (gsplat, nvblox, neural reconstruction) and refactors VizSession's frame loop so the poses fed to renderers match the poses submitted to OpenXR.
ProjectionLayer
In-loop contract:
The runtime / CloudXR paces the app via xrWaitFrame; if the renderer takes 30 ms, the app runs at ~30 fps and the runtime compositor reprojects the last submitted frame at display rate.
Storage: per-slot (color, depth) DeviceImages (7-slot mailbox per QuadLayer's pattern). Fragment shader samples color + depth and writes gl_FragDepth so QuadLayer / future OverlayLayer Z-composite correctly in the shared RT. Stereo: paired (left, right) slot storage; submit() ships both eyes on one CUDA stream.
In kXr, a visible ProjectionLayer that doesn't submit for the current frame is skipped at record() time so stale RGBD never gets composited under a new projection-layer pose. The freshness gate is off in kWindow / kOffscreen where no XR pose mismatch is possible.
Single XrCompositionLayerProjection covers QuadLayers + ProjectionLayer together — CloudXR fast-paths when only ProjectionLayer is present, and squashes per-layer-timewarped when mixed (Monado's compute-shader layer accumulator handles this automatically).
VizSession frame-loop refactor
VizSession::begin_framepreviously returned placeholder identity views; the real per-eye XR poses were only acquired later insideVizCompositor::renderviabackend_->begin_frame. That meant renderers callingsubmitagainst the returned FrameInfo were rendering against the wrong pose.Moves backend frame acquisition into
VizSession::begin_frame: acquired Frame is stored incurrent_backend_frame_and consumed byend_frame.FrameInfo.viewsnow carries the real xrLocateViews output, so the in-loop pattern above is pose-correct by construction.VizCompositor::render takes the acquired Frame as a parameter; the existing FrameGuard still owns end_frame/abort_frame protocol balance.
Adds Frame::predicted_display_time_ns so XR's
last_frame_state_.predictedDisplayTimeflows through toFrameInfo.predicted_display_time.LayerBase::on_frame_begin
New virtual called from VizSession::begin_frame on every registered layer (default no-op). ProjectionLayer uses it to clear its submitted-this-frame freshness flag.
Minor changes
DeviceImageallows PixelFormat::kD32F (was rejected as "reserved for ProjectionLayer"). kD32F + mip_levels > 1 rejected explicitly.ViewInfoandFrameInfo.views. NoProjectionViewSnapshot/predicted_views/get_last_frame_info— the renderer readsinfo.viewsreturned bybegin_frame.Tests
Summary by CodeRabbit
Release Notes
New Features
Enhancements