10 Replies
      Latest reply on Sep 19, 2019 4:13 AM by ltxitai
      Carniphage Level 1 Level 1 (15 points)

        I have an idea for a rendering method which I thought would be a good use for SceneKit and Metal.

        I have figured out how to do DRAW_QUAD and DRAW_SCENE shaders which work.

         

        But I am repeatedly hitting problems with a lack of documentation.

         

        I want a scene shader technique to render-out a special buffer.  This shader needs some custom uniform parameters for each draw call. My technique assocates this uniform with a symbol and SCNTechnique provides a handy callback solution:  handleBindingOfSymbol( ) -

        A block is invoked at each draw call allowing my code to supply custom pararameters ....

         

        ... Except this is an OpenGL-only call.  The block is never called for a Metal implementation.   There is no direct Metal equivalent.

        Is there a way round this? Or should I just leave Metal alone for a year?

        • Re: SCNTechnique & Metal  handleBindingOfSymbol()
          Xartec Level 1 Level 1 (0 points)

          2 years later... the Metal equivalent is https://developer.apple.com/documentation/scenekit/scnprogram/1524047-handlebinding but unlike handleBindingsOfSymbol it’s only supported by SCNProgram and not SCNTechnique. This, as well as not being able to pass (device space) MTLBuffers to a SCNProgram, poses serious limitations on the SceneKit+Metal combination.

            • Re: SCNTechnique & Metal  handleBindingOfSymbol()
              knl Level 1 Level 1 (0 points)

              Have you found a way to pass per-frame data to an SCNTechnique Metal shader by now? For ray-tracing I need the current `pointOfView` available in the shader, but can't find a way to access it... any ideas?

                • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                  gc_arkit Level 1 Level 1 (0 points)

                  Yes there is a way to pass symbols to metal shaders used in an SCNTechnique pass.  Please respond to this message if you still need to know and I can post it.  It’s fairly a fairly involved method.

                    • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                      knl Level 1 Level 1 (0 points)

                      Hi gc_arkit, I would indeed be interested in your solution to this issue!

                        • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                          gchiste Apple Staff Apple Staff (170 points)

                          Hello knl,

                           

                          To pass a custom symbol to a metal shader used in an SCNTechnique pass:

                           

                          Step 1 - Defining a custom symbol in an SCNTechnique:

                          //…other dictionaries to define a technique,
                                "symbols": [
                                          //…other symbols,
                                          "mvpSymbol": ["semantic": "modelViewProjectionTransform"],
                                          "myFloatSymbol": ["type": "float"]
                                          //…other symbols
                                      ],
                          //…other dictionaries to define a technique

                           

                          Important notes for Step 1:

                           

                          The name that you choose for your symbol here MUST correspond with the name you use in your pass definition (which we will define next).  Please see the SCNTechnique page for a list of semantics and types.

                           

                          Step 2 - Adding your custom symbols to a particular pass:

                           

                          //…rest of pass definition
                          "inputs": [
                              "symbolAsCalledInMetal": "mvpSymbol",
                               "myFloat": "myFloatSymbol"
                          ],
                          //…rest of pass definition

                           

                          Important notes for Step 2:

                           

                          Whatever you decide to name your symbol for a particular pass, THAT is the name that you must use for the variable in your struct in Metal (which we will define soon).  You can use the same symbol in different passes, just make sure that you add it to the "inputs" of each pass.

                           

                          Step 3 - Defining the metal shader for a particular pass:

                           

                          I think you probably already have this step down, but just in case, here is an example where we assume we have a .metal file with a vertex shader named "myVertexShader", and a fragment shader named "myFragmentShader"

                           

                          //...rest of pass definition
                          "metalVertexShader": "myVertexShader",
                          "metalFragmentShader": "myFragmentShader",
                          //...rest of pass definition

                           

                           

                          Step 4 - Binding a value to each of your custom symbols on a per-frame basis:

                           

                          //...Called inside the renderer(_:updateAtTime:) method
                          // For "myFloatSymbol", use NSNumber and pass the number that you want to use in your shader
                          sceneView.technique?.setObject(NSNumber(value: 3.0), forKeyedSubscript: “myFloatSymbol” as NSCopying)
                          // For "mvpSymbol", use NSValue and pass it the transform that you want to use in your shader
                          sceneView.technique?.setObject(NSValue(scnMatrix4: transform, forKeyedSubscript: “mvpSymbol” as NSCopying)

                           

                          Important notes for Step 4:

                           

                          I recommend that you bind your values in the renderer(_:updateAtTime:) method (assuming you want to update your symbols on a per-frame basis).  Using the setObject method of an SCNTechnique, you want to use either NSNumber or NSValue depending on the type of your symbol, in this example, I use SCNMatrix4 for the transform because that is the type that corresponds with float4x4 in Metal, and Model-View-Projection Transforms are 4x4 matrices.  Lastly, note that I am using the name of the symbol defined in the technique's "symbols" dict here, not the name of the symbol defined in a particular pass' "inputs" dict.

                           

                          Step 5 - Defining your struct in your .metal file:

                           

                          typedef struct {
                             // other symbols...
                               float myFloat;
                              float4x4 symbolAsCalledInMetal;
                             // other symbols…
                          } Inputs;

                           

                          Important notes for Step 5:

                           

                          Notice that in the struct definition, I am using the names defined for a particular pass' "inputs" dict.

                           

                          At this point, you should now be able to access your custom symbols in your .metal shaders!

                            • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                              vade Level 1 Level 1 (10 points)

                              Hello - is it possible to use this for an arbitrary sampler2D that does not reference an on disk image?

                               

                              TL;DR:

                              How does one reference a 'not on disk' sampler2D symbol and pass it to an SCNTechnique? My technique works if I reference an image from my bundle, but if I don't, I cannot find a way to pass in an existing id<MTLTexture> to the sampler symbol my technique has set up.

                               

                              Long:

                              I have a valid working SCNTechnique which uses a custom sampler2D symbol as an input to my metal fragment pass. I am attempting to pass in an external (not from Scenekit) id<MTLTexture>that I get from a hardware sensor as input in a post process pass.

                              Following the SCNShadable docs which state an id<MTLTexture> can be passed as a shader input via an SCNMaterialProperty which has the proper contents set. This 100% works in a shader modifier pass - but fails for SCNTecnique!

                              For a SCNTechnique, I get error logs indicating "No Storage for texture" and the Metal GPU frame capture indicates there is a default 4x4 pixel white texture set for the sampler (presumably from the SCNTecnique?). However, I've been able to validate that my custom id<MTLTexture> is valid and has contents in the debugger - its format, width, height and contents all are as expected, I just can't seem to reference an arbitrary texture into a scene kit technique pass correctly.

                               

                              Has anyone gotten something like this to work?

                              Thank you.

                                • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                                  gchiste Apple Staff Apple Staff (170 points)

                                  Hello Vade,

                                   

                                  When passing a symbol with type sampler2D in an SCNTechnique, the value of that symbol must be an image located in the application bundle.

                                    • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                                      ltxitai Level 1 Level 1 (0 points)

                                      @gchiste - Does the same hold true for Shader Modifiers? No matter what I try, I can't seem to pass an `id<MTLTexture>` to a shader modifier - either it makes the entire render pipeline invalid, or else it gets called with an 4x4 all-white "default SCN texture".

                                      Is there some documentation on the limitations of what one can pass to a Shader Modifier when using Metal? (the current official documentation is useless, frankly, as it only refers to OpenGL - in Metal one wouldn't pass a texture as a `sampler` but as a `texture2d`...).

                                        • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                                          gchiste Apple Staff Apple Staff (170 points)

                                          Have you tried setting your MTLTexture as the contents of an SCNMaterialProperty, and then passing that SCNMaterialProperty as a texture2d?

                                            • Re: SCNTechnique & Metal  handleBindingOfSymbol()
                                              ltxitai Level 1 Level 1 (0 points)

                                              Thank you for the quick reply!

                                              I tried it before, and every time I did the render pipeline completely failed to execute, claiming the pipeline state was null.

                                              However, I tried it again - and it just worked! I tried to reproduce what I did earlier, but couldn't get the same error.

                                              I can only guess there is some tricky bit, or obscure thing I did wrong before, but couldn't decipher SceneKit's error messages.

                                              Do you have any guess as to what could have caused the render pipeline state to be null before whenever I tried to set SCNMaterialProperty with an id<MTLTexture> contents?

                                               

                                              Thanks again for the help.