Camera nodes have sizing dimension input, Perspective camera has FOV

[tobyspark]

I like QC’s unit system of -1,+1 with square units. But which dimension? QC used width. FOV is typically calculated on height. It looks like in Fabric the perspective camera uses horizontal and orthographic camera uses vertical?

This PR adds Sizing Dimension to both cameras, so this can be adopted on a case-by-case basis. On a practical note, fixing height and extending width per aspect is really handy in performance contexts where you can design for a single 16:9 output, and output your 3D world in wider-and-wider-wide-screen by adding screens to the side. Oh, you’ll want to increase the field of view then too... so that’s now a perspective camera input too.

Note - I’ve run out of time so this is currently untested! Will get back to this... at some unknown point soon.

[vade]

i think we should standardize on width , but i havent yet tacked the nuance of

• how do think about the coordinate system when the camera has moved from the standard FOV + position?

What ive done now is

• if you delete the camera, you get QC's perspective camera -width - +width and -aspect - +aspect unit coords

• if you instantiate a camera, you get the cameras view inside of a world of -width - +width

what ive yet to figure out, or systematize is how the unit -> pixels pixels -> unit conversions happen when the coordinate system is dynamic based off of moving cameras etc.

my concerns is maybe cameras need to be refactored into subpatch / macro which make it very clear to any parent node what the coordinate system is.

Im honestly open to suggestions here, because i think its quite nuanced really and theres a lot of valid approaches each with a ton of side effects. Lets discuss!

[tobyspark]

I used to think standardising on width was good (in short, QC), but I REALLY REALLY have experienced the benefit of sizing on height. I don’t see a good reason to hold back that choice. I think it’s project dependent.

Beyond that... yeah. I think the basic idea of -1, +1 is good, and yes there’s nuance as to whether that’s at the camera default or current setting or...? That said, if I’m changing the FOV of the perspective camera, I wouldn’t expect the whole scene to scale up, I’d expect to see more of it. So that would be width=2 at the “default” FOV?

You might be right about subpatch. Though I think the ability to not have a camera in the root patch is a good “progressive disclosure” feature. (progressive complexity? whatever that phrase is)

[tobyspark]

I have clarified this PR by splitting the commit. There is now a fix to GraphRenderer’s resize function, with documentation on expected behaviour. And then the additions to Camera spec.

[tobyspark]

The default camera setup wasn’t consistent with the camera nodes. That is now fixed.

I think this is good.

• Default is QC behaviour
• Progressive disclosure by adding a camera node, which if perspective doesn’t change anything but exposes options to the current/default setup.
• Perspective camera has FOV and sizing dimension inputs
• Orthographic camera has sizing dimension inputs

[tobyspark]

what ive yet to figure out, or systematize is how the unit -> pixels pixels -> unit conversions happen when the coordinate system is dynamic based off of moving cameras etc.

I don’t think you can? In that QC couldn’t really either, it was only valid at the origin? I think if you care about pixel precision, you’re going to use an ortho camera, and Fabric provides that (and this PR allows you to set world units to pixels via the size input). Or if you’re doing 2.5D stuff, you can roll your own conversion with the Rendering Info node.

[tobyspark]

my concerns is maybe cameras need to be refactored into subpatch / macro which make it very clear to any parent node what the coordinate system is.

Am I thinking about this right – There can only be one active camera per scene / render target. So there needs to be a way of ensuring there is one, and only one, active camera in the graph hierarchy, or sub-hierarchies of Render to Image and Depth. And logically it doesn’t actually matter where the camera is specified in each hierarchy.

If that is true, I think there are three reasonable options

1. The root graph has published inputs to specify the camera, as does Render to Image and Depth
2. Implicit default camera, add a camera to override, cannot add further camera. Goes for root graph and any Render to Image subgraphs. But would the override cascade down to become the new implicit default?
3. Enforced camera at root level of each document graph and any Render to Image subgraphs.

Or, is there an approach where there is effectively “no” root camera – ortho, 2D-like – and cameras are subgraph nodes which apply a (perspective) transform to the child graph? The math broadly works, but the finer points of rendering don’t though, right?

[vade]

Yea, ive struggled with that question tbh, because in theory, you can have multiple cameras, (think of 4 up quad rendering into a single window in CAD software, ie multiple points of view) etc, and that strictly speaking would require each camera being renderered to a sub view port of the existing window.

Another option is having cameras respect layer orders, so multiple camera's could be instantiated making weird output on the same composite image (different geo with different FOV and angles composited together)

Honestly thats more trouble than its worth , but fun in theory and worth thinking about architecture wise, but at some point i think youd say 'Fabric should pick a paradigm' and i agree with that, and think you are right.

Let me consider your 3 options a bit!

[tobyspark]

To clarify for myself, scenes ≠ render targets, per the 4-up rendering example. The root graph can only have one render target: "the screen” lets say. But a Render in Image node could potentially have multiple outputs.

Also, there’s an obvious reason to go for option 2 or 3 – cameras as nodes – as this means you can patch to the inputs in the root graph.

[vade]

The problem with cameras as nodes / subgraphs is you have to duplicate the scene into the nodes. if you wanted a 4up of the same content, youd duplicate the graph, which is very inefficient

[tobyspark]

...that’s what Render to Image could get you, though. One graph/scene, multiple render targets / images.

[tobyspark]

I’m also going to note this comment again, as it is me changing my mind without flagging it clearly.

That said, if I’m changing the FOV of the perspective camera, I wouldn’t expect the whole scene to scale up, I’d expect to see more of it. So that would be width=2 at the “default” FOV?

The 2D-like QC default is good, but we shouldn’t hold that when changing FOV. I think the current implementation does. I think we just have to pick an arbitrary perspective sizing, i.e. with the camera at a set distance from the origin, what the “units” across (or up) at the origin is a set arc-angle striking the origin. Those set values would align with the default camera, i.e. QC default.

[vade]

just for clarity terms, today in Satin:

* Render Order - publicaly set on an Object - the order with which objects (ie canonical Satin.Objects are rendered. Objects within a specific layer are sorted by order if sorting is enabled, which it is by default.

• Render Layer - an internal grouping of objects rendered in render order (if sorting is enabled) together within a pass, based off of material needs (blending mode), opaque, transparent, overlay etc

• Render Pass - one encoding of render commands to render / rasterize a single layer (or sometimes multiple layers depending on renderer mode) to pixels with a specific configuration

• Render Mode (new in Satin 2.0) .forward, .forwardPlus, .deferredGeometry - directly governs how many passes the encoder produces for the same scene — a .deferredGeometry scene gets a geometry pass + deferred lighting pass before the unlit and transparent forward passes.

• Render Target(s) - image(s) associated with rendering content as part of a single render pass, could be depth, color, albedo, normals, velocity map, etc. When rendering a single pass, deferred rendering uses multiple render targets (anything in addition to color, depth, stencil is referred to as multiple render target (MRT) or g-buffer rendering)

• Viewport - the space in the raster of a render target that rendering outputs to. A render target could be rendered to multiple times over different viewports with different sizes and origins. We dont currently leverage or expose viewport as a camera variable externally, though Satin does support it on Renderer

• Vertex Amplification - a technique to allow rendering the same geo in a single pass to multiple distinct targets or even viewports - satin does not support the latter, it requires shaders to use [[viewport_array_index]] but satin support vertex amplification to different targets, but it requires context configurations and is a paradigm choice.

internally theres some additional bullshit to make it work, but thats the gist. Today in Satin 1.0 as a forward renderer, theres typically a single pass with all layers. In Satin 2.0, to support additional features modern renderers support, we carve up layers into passes and render them / composite them as needed.

...that’s what Render to Image could get you, though. One graph/scene, multiple render targets / images.

Render to Image does that in Satin 2.0 and for a single pass output a collection of images.

Options for rendering multiple cameras as i see it is going to be governed by Satin's underlying paradigm:

• Any Graph - Single pass - Single Active Camera paradigm

  • Users could add multiple cameras and control them / position / look at whatever
  • Only one is active at a time, enforced by the runtime
  • Active one can change via graph logic to allow changing ortho, perspective, what have you for creative reasons.
  • this is the most straightforward with Satin's rendering pipeline today with the least amount of fuckery

• Render in Image: one set of outputs per camera, multiple render passes produces n sets of images (multi - MRT)

  • This would be very very expensive, and i think not a great UI to work with imo
  • You could do something in the root graph that only works in a specific sub graph, which to me is a code smell
  • I think im sour on this approach as it makes camera handling a special case, which feels mighty akward to me.

• Any graph: Single pass, support multiple cameras, each camera gets its own viewport

  • Each camera just renders the scene into the current pass
  • Cameras could be enabled or disabled,
  • Satins renderer allows for multiple cameras to be passed into a single draw command to render via vertex amplification, allowing the same geo to be rendered multiple times to different output:

public func draw(renderPassDescriptor: MTLRenderPassDescriptor, commandBuffer: MTLCommandBuffer, scene: Object, cameras: [Camera], viewports: [MTLViewport], viewMappings: [MTLVertexAmplificationViewMapping] = [], renderTarget: MTLTexture) {

from the Example MultipleViewportRenderer:

  subRenderer.draw(
            renderPassDescriptor: subRenderPassDescriptor,
            commandBuffer: commandBuffer,
            scene: subScene,
            cameras: [subCamera0, subCamera1],
            viewports: [subViewport, subViewport]
        )

 - This doesnt play nice however with some other techniques, and requires the context to be configured for it, and requires materials to be updated for it - its sort of a 'fundamental choice' 

Im not sure if theres any other obvious wins after doing a review of Satins rendering architecture tbh

My feeling is a single active camera, but allow additional camera's to be placed in a scene.

• Default Camera (no camera node, default coordinates)
• Add a Camera node - override default, move camera and FOV etc etc, now you are in control of what the screen space coordinates are
• Add multiple cameras - for a single frame / execution step only one can be active, but this is configurable in the UI allowing more complex setups to happen

This means no multi viewport shit, which it think is fine.

[tobyspark]

I’m not so sour on Render to Image being capable of having multiple active cameras, as it feels to me quite natural that its ability to have multiple image outputs already expresses the requirement of cameras being paired to outputs without introducing new concepts or twisting existing ones. (I’m curious why it would be “very very expensive” – is that because it would require vertex amplification?)

But, I also feel that’s a bonus feature we can happily push down the road. Similarly, if instead we wanted to expose viewport on a camera node, and have multiple active cameras, that’s something we can push down the road. All as long as whatever we chose now doesn’t hamper us with either of those directions.

i.e. I agree – “no multi viewport shit”, perhaps not forever, but definitely not for now.

I have a very strong design opinion on your last point though: a multiple camera system where one is managed to be active is the wrong abstraction. Conceptually, there is only one camera active at a time, and it can be instantaneously set to any spec. That is what the primitive should be, and the patching around it should be a way of “puppeting” where it should be etc. at any one moment. Practically, that means

• Fabric users build up a generalisable system for positioning, animation etc. that can apply to anything in 3D space, not just cameras (example of which is the transform array patches I’ve got in *spark stage for spatial arrangements of meshes)
• If you want a studio-like multi-camera system, you’ve got the primitives to make any spin you want on that.
• The multi-camera system breaks down the minute you want to do anything “between two cameras” that isn’t a hard cut. i.e. there’s a whole raft of things you might want to do with a camera that end up with you ditching the multi-camera system anyway.
• I also think it’s cleaner for the limit of one “active” camera to be represented by the presence of a camera node, rather than shared state between a number of nodes that isn’t visible at the graph level to the user.
• If there is going to be a world of multiple cameras for a multi-output setup, you’ve now got two dimensions of multiple cameras to reason about and craft a UX for. It’s unnecessary.

The one place I see where that breaks down is switching between perspective and ortho cameras – I’m not sure if they are fundamentally different types in Satin, or whether it’s just a matter of transform matrices etc.

[tobyspark]

From discussion on a call – Adopt the paradigm that a graph has a camera node as a fixed feature. i.e. you cannot add another, you cannot delete it. It might house other one-per-graph aspects such as clear colour. Rationale of that vs. a multiple-cameras-in-the-scene with some-new-means-to-make-one-exclusively-active in the post above.