Using stencil buffers with SCNTechnique

For the past two days I've been trying to use stencil buffers with SCNTechnique. I feel like I've tried everything and I still have a lot of open questions. Usually I gather information from other people's questions, but posts on this topic appear few and far between. I hope someone here can point me in the right direction.

Running on macOS 10.15.3, Xcode 11.3.1, using Metal. Using basic SceneKit template, modified slightly.

I'm trying to achieve a portal effect with a couple of passes:

1. Render a single plane that acts as the portal, into a stencil buffer.
2. Render the inside of the portal, clipped by the stencil buffer.
3. Render the outside of the portal.
4. Mix the two color + depth buffers together (using a simple custom Metal shader).

For steps 1-3 I don't need a custom program/shaders, for step 4 I've written a custom shader that mixes two inputs.

There's three objects in my scene: ship (bitmask 8), plane (bitmask 4), sphere (bitmask 2). The sphere shold only be visible through the stencil of the plane, like a portal.

From what I've learned from SCNTechnique documentation and OpenGL paradigms, the stencil buffer offers the functionality I'm looking for.

The technique as attached below does not pass Metal API validation, complaining that "validateDepthStencilState:3857: failed assertion `MTLDepthStencilDescriptor uses frontFaceStencil but MTLRenderPassDescriptor has a nil stencilAttachment texture'".

I can change "stencilStates.enable" to `false` in the "inner" pass to at least make it run, but it appears the the stencil buffer is not fed correctly into the "inner" pass and cropping/stencilling does not occur. When I look at the GPU frame capture, the "portal_stencil" pass output is marked with a purple exclamation mark.

Since the "inner" pass does not need to write to the stencil buffer (only test against it), I think "stencilStates.enable" should be `false`, right?

(Q) How do I get the "inner" pass to respect/use the stencil buffer? Do I even need separate stencil buffer, or should I be able to use the default stencil buffer?

This technique does use a stencil buffer but does not specify its own inside `targets` like I'm attempting. It uses custom shaders for every pass, which I'd like to avoid unless it's necessary. I haven't found yet how to write a passthrough shader in Metal (a shader that follows default behavior).

Conceptually, I think should also be able to render the "inner" pass immediately into the "outer" color buffer while leveraging depth testing (against outer buffer) and stencil testing (against portal buffer). So [outer -> portal_stencil -> inner]. That would reduce the draw passes to three. (Q) Am I thinking the right way?

Thanks a lot in advance, I'm really struggling with this.

{
  "passes": {
    "outer": {
      "draw": "DRAW_SCENE",
      "outputs": {
        "color": "color_outer"
      },
      "excludeCategoryMask": 6
    },
    "portal_stencil": {
      "draw": "DRAW_SCENE",
      "outputs": {
        "stencil": "stencil_scene"
      },
      "stencilStates": {
        "clear": true,
        "enable": true,
        "behavior": {
          "depthFail": "zero",
          "fail": "zero",
          "pass": "replace",
          "function": "always",
          "referenceValue": 255,
          "writeMask": 1
        }
      },
      "includeCategoryMask": 4,
      "excludeCategoryMask": 10
    },
    "inner": {
      "draw": "DRAW_SCENE",
      "inputs": {
        "stencil": "stencil_scene"
      },
      "outputs": {
        "color": "color_inner"
      },
      "colorStates": {
        "clear": true
      },
      "stencilStates": {
        "enable": true,
        "behavior": {
          "depthFail": "keep",
          "fail": "keep",
          "pass": "keep",
          "function": "equal",
          "referenceValue": 255,
          "readMask": 1,
          "writeMask": 0
        }
      },
      "includeCategoryMask": 2,
      "excludeCategoryMask": 12
    },
    "mix": {
      "draw": "DRAW_QUAD",
      "metalVertexShader": "mixVertex",
      "metalFragmentShader": "mixFragment",
      "inputs": {
        "colorInner": "color_inner",
        "colorOuter": "color_outer"
      },
      "outputs": {
        "color": "COLOR"
      }
    }
  },
  "sequence": [
    "portal_stencil",
    "inner",
    "outer",
    "mix"
  ],
  "targets": {
    "color_outer": {
      "type": "color"
    },
    "color_inner": {
      "type": "color"
    },
    "stencil_scene": {
      "type": "stencil"
    }
  },
  "symbols": {
    
  }
}