SCNTechnique & Metal handleBindingOfSymbol()

I have an idea for a rendering method which I thought would be a good use for SceneKit and Metal.

I have figured out how to do DRAW_QUAD and DRAW_SCENE shaders which work.


But I am repeatedly hitting problems with a lack of documentation.


I want a scene shader technique to render-out a special buffer. This shader needs some custom uniform parameters for each draw call. My technique assocates this uniform with a symbol and SCNTechnique provides a handy callback solution: handleBindingOfSymbol( ) -

A block is invoked at each draw call allowing my code to supply custom pararameters ....


... Except this is an OpenGL-only call. The block is never called for a Metal implementation. There is no direct Metal equivalent.

Is there a way round this? Or should I just leave Metal alone for a year?

Replies

2 years later... the Metal equivalent is https://developer.apple.com/documentation/scenekit/scnprogram/1524047-handlebinding but unlike handleBindingsOfSymbol it’s only supported by SCNProgram and not SCNTechnique. This, as well as not being able to pass (device space) MTLBuffers to a SCNProgram, poses serious limitations on the SceneKit+Metal combination.

Have you found a way to pass per-frame data to an SCNTechnique Metal shader by now? For ray-tracing I need the current `pointOfView` available in the shader, but can't find a way to access it... any ideas?

Yes there is a way to pass symbols to metal shaders used in an SCNTechnique pass. Please respond to this message if you still need to know and I can post it. It’s fairly a fairly involved method.

Hi gc_arkit, I would indeed be interested in your solution to this issue!

Hello knl,


To pass a custom symbol to a metal shader used in an SCNTechnique pass:


Step 1 - Defining a custom symbol in an SCNTechnique:

//…other dictionaries to define a technique,
      "symbols": [
                //…other symbols,
                "mvpSymbol": ["semantic": "modelViewProjectionTransform"],
                "myFloatSymbol": ["type": "float"]
                //…other symbols
            ],
//…other dictionaries to define a technique


Important notes for Step 1:


The name that you choose for your symbol here MUST correspond with the name you use in your pass definition (which we will define next). Please see the SCNTechnique page for a list of semantics and types.


Step 2 - Adding your custom symbols to a particular pass:


//…rest of pass definition
"inputs": [
    "symbolAsCalledInMetal": "mvpSymbol",
     "myFloat": "myFloatSymbol"
],
//…rest of pass definition


Important notes for Step 2:


Whatever you decide to name your symbol for a particular pass, THAT is the name that you must use for the variable in your struct in Metal (which we will define soon). You can use the same symbol in different passes, just make sure that you add it to the "inputs" of each pass.


Step 3 - Defining the metal shader for a particular pass:


I think you probably already have this step down, but just in case, here is an example where we assume we have a .metal file with a vertex shader named "myVertexShader", and a fragment shader named "myFragmentShader"


//...rest of pass definition
"metalVertexShader": "myVertexShader",
"metalFragmentShader": "myFragmentShader",
//...rest of pass definition



Step 4 - Binding a value to each of your custom symbols on a per-frame basis:


//...Called inside the renderer(_:updateAtTime:) method
// For "myFloatSymbol", use NSNumber and pass the number that you want to use in your shader
sceneView.technique?.setObject(NSNumber(value: 3.0), forKeyedSubscript: “myFloatSymbol” as NSCopying)
// For "mvpSymbol", use NSValue and pass it the transform that you want to use in your shader
sceneView.technique?.setObject(NSValue(scnMatrix4: transform, forKeyedSubscript: “mvpSymbol” as NSCopying)


Important notes for Step 4:


I recommend that you bind your values in the renderer(_:updateAtTime:) method (assuming you want to update your symbols on a per-frame basis). Using the setObject method of an SCNTechnique, you want to use either NSNumber or NSValue depending on the type of your symbol, in this example, I use SCNMatrix4 for the transform because that is the type that corresponds with float4x4 in Metal, and Model-View-Projection Transforms are 4x4 matrices. Lastly, note that I am using the name of the symbol defined in the technique's "symbols" dict here, not the name of the symbol defined in a particular pass' "inputs" dict.


Step 5 - Defining your struct in your .metal file:


typedef struct {
   // other symbols...
     float myFloat;
    float4x4 symbolAsCalledInMetal;
   // other symbols…
} Inputs;


Important notes for Step 5:


Notice that in the struct definition, I am using the names defined for a particular pass' "inputs" dict.


At this point, you should now be able to access your custom symbols in your .metal shaders!

Hello - is it possible to use this for an arbitrary sampler2D that does not reference an on disk image?


TL;DR:

How does one reference a 'not on disk' sampler2D symbol and pass it to an

SCNTechnique
? My technique works if I reference an image from my bundle, but if I don't, I cannot find a way to pass in an existing
id<MTLTexture>
to the sampler symbol my technique has set up.


Long:

I have a valid working

SCNTechnique
which uses a custom sampler2D symbol as an input to my metal fragment pass. I am attempting to pass in an external (not from Scenekit)
id<MTLTexture>
that I get from a hardware sensor as input in a post process pass.

Following the

SCNShadable
docs which state an
id<MTLTexture>
can be passed as a shader input via an
SCNMaterialProperty
which has the proper contents set. This 100% works in a shader modifier pass - but fails for
SCNTecnique
!

For a

SCNTechnique
, I get error logs indicating "No Storage for texture" and the Metal GPU frame capture indicates there is a default 4x4 pixel white texture set for the sampler (presumably from the
SCNTecnique
?). However, I've been able to validate that my custom
id<MTLTexture>
is valid and has contents in the debugger - its format, width, height and contents all are as expected, I just can't seem to reference an arbitrary texture into a scene kit technique pass correctly.


Has anyone gotten something like this to work?

Thank you.

@gchiste - Does the same hold true for Shader Modifiers? No matter what I try, I can't seem to pass an `id<MTLTexture>` to a shader modifier - either it makes the entire render pipeline invalid, or else it gets called with an 4x4 all-white "default SCN texture".

Is there some documentation on the limitations of what one can pass to a Shader Modifier when using Metal? (the current official documentation is useless, frankly, as it only refers to OpenGL - in Metal one wouldn't pass a texture as a `sampler` but as a `texture2d`...).

Have you tried setting your MTLTexture as the contents of an SCNMaterialProperty, and then passing that SCNMaterialProperty as a texture2d?

Thank you for the quick reply!

I tried it before, and every time I did the render pipeline completely failed to execute, claiming the pipeline state was null.

However, I tried it again - and it just worked! I tried to reproduce what I did earlier, but couldn't get the same error.

I can only guess there is some tricky bit, or obscure thing I did wrong before, but couldn't decipher SceneKit's error messages.

Do you have any guess as to what could have caused the render pipeline state to be null before whenever I tried to set SCNMaterialProperty with an id<MTLTexture> contents?


Thanks again for the help.

Is setObject(_:forKeyedSubscript:) safe to call every frame? The documentation appears to suggest otherwise:

Use this method when you need to set a value infrequently or only once. To update a shader value every time SceneKit renders a frame, use the handleBinding(ofSymbol:using:) method instead.

I am using ARKit + Scenekit and I am trying to pass in the smoothedSceneDepth from the LiDAR Scanner into a custom SCNTechnqiue. I convert the CVPixelBuffer into a MTLTexture, then pass that into my technique like so:

Code Block
let mtlTex = PixelBufferToMTLTexture(pixelBuffer: pixelBuffer)
arscnView.technique?.setObject(SCNMaterialProperty(contents: mtlTex), forKeyedSubscript: "camera_depth" as NSCopying)


This appears to work correctly, however I have had the occasional crash when calling setObject every frame.