Masking a UIVisualEffectView in iOS 11

Let's say you have the CGPath for masking ready to use and stored in the path variable

    let maskLayer = CAShapeLayer()
    maskLayer.path = path.cgPath
    maskLayer.fillRule = kCAFillRuleEvenOdd // Could be different as you need
   
    if #available(iOS 11.0, *) {
      blurView.layer.mask = maskLayer
    }
    else {
      let maskView = UIView(frame: bounds)
      maskView.backgroundColor = UIColor.black
      maskView.layer.mask = maskLayer
      blurView.mask = maskView
    }

Then this should work for blurView to get a mask effect on both iOS 10 & 11.

The problem is that you are trying to use nested masking. I refer to the comment on CALayer.mask:

/ A layer whose alpha channel is used as a mask to select between the
* layer's background and the result of compositing the layer's
* contents with its filtered background. Defaults to nil. When used as
* a mask the layer's `compositingFilter' and `backgroundFilters'
* properties are ignored. When setting the mask to a new layer, the
* new layer must have a nil superlayer, otherwise the behavior is
* undefined. Nested masks (mask layers with their own masks) are
* unsupported. */

Specifically by setting up your mask as "maskView.layer.mask = maskLayer; effectView.maskView = maskView" you've created a nested mask as of iOS 11. On iOS 10 we would snapshot 'maskView' and you wouldn't have the same problem.

You need to supply 'maskLayer' directly, rather than indirectly via maskView, which you can do by overriding UIView.layerClass() to return CAShapeLayer.self, then configuring an instance of your subclass's layer as you already are. Then you will have only a single level of masking.

Alternatively you *can* use effectView.layer.mask as of iOS 11 and get the correct result if you want to, but then you have to write code specifially for iOS >=11 vs iOS >11. If you do the override as I recommend above you can use the same code on both (modulo live masking, which is a feature of the iOS 11 implementation).

Could you elaborate on what needs the custom layerClass override? Is it the maskView like i've done below, or is it the visualEffectsView?

final class ShapeLayerView: UIView {
    var shapeLayer: CAShapeLayer {
        return layer as! CAShapeLayer
    }
   
    override class var layerClass: AnyClass {
        return CAShapeLayer.self
    }
}

let path = UIBezierPath(rect: view.frame)
path.append(UIBezierPath(rect: self.viewPortView.frame).reversing())
      
let shapeLayer = CAShapeLayer()
shapeLayer.path = path.cgPath
shapeLayer.fillColor = UIColor.white.cgColor
      
let maskView = ShapeLayerView(frame: view.frame)
maskView.shapeLayer.path = path.cgPath
maskView.shapeLayer.fillColor = UIColor.white.cgColor
      
visualEffectsView.mask = maskView

Yes, your mask view needs the override. The code you have above seems reasonably correct (I'm not certain how many of those frame calls you have should be bounds calls, if any, but thats the only obvious thing I see).