How much work do you queue in advance? Can you have at least 30ms of GPU work queued at any given time for instance?

hi, Could you share a full and self-contained example project? ideally zipped or hosted I happen to have almost the same hardware configuration and could give a quick look at it. In the meantime I recommend profiling your program with Instruments (Metal one) to see what happens.

I expect what you're seeing is your flags being applied to all shaders while they should only be applied to the Core Image ones. https://developer.apple.com/videos/play/wwdc2020/10021/ describes a way to apply the Core Image flags to Core Image shaders only, through custom build rules. Alternatively you can add your CI shaders in a distinct target.

if the regression calculation in Metal takes too long After how long do you see it failing? Do you also have a requirement to hardcode the threadgroup size? Or did you notice that it performed best with 16x16 ? https://developer.apple.com/documentation/metal/compute_passes/calculating_threadgroup_and_grid_sizes provides sample code for dynamically computing this size. As a side note, although I don't expect this to be the core of your issue, you shouldn't instantiate the compute pipeline state as part of the process() function. This needs to be done only once, not for each process() call. This state should be created beforehand or at least cached.

In Xcode you can define a symbolic breakpoint on -[CAMetalLayerDrawable texture] to break each time this method is called (including the valid calls). This can help you find where this is used in your app.

Did you profile to see what was the bottleneck in the CoreImage implementation and where the bottleneck is now?

Your mtkView size is 414x233 points, which is 828x466 pixels with 2x display scaling. By default, the drawable texture size in MTKView follows the view size, which makes sense to render content at the native resolution and perfectly fitting the UIView bounds. So you have two options: force a specific drawable size on MTKView (disable MTKView.autoResizeDrawable and set MTKView.drawableSize to whatever you want). Texture will have the correct size but it may look weird on display. render to a MTLTexture that you create and own (thus with the size that you want) instead of using the one from the drawable

The only idea that comes to my mind (without knowing the actual transform to do, since it's definitely not uniform) would be to try to use CIPerspectiveTransform.

Why are you using a CAMetalLayer if you don't use any feature from Metal? Your example looks like a very uncommon usage to me: in my experience you get red output when the framebuffer isn't even cleared because there is no render pipeline (MTLRenderCommandEncoder) scheduled to render into that drawable CAMetalLayer provides drawable that cannot be written to by default, only rendered into, at least from Metal pipeline perspective. Is your texture storage even shared? I would expect the texture created by CAMetalLayer to have private storage, so you can only write to it through Metal commands.

Your kernel takes a uint* result. uint is 4 bytes, but your Swift code gives a buffer of 3 UInt8 aka 3 bytes.

The problem is I can't redraw the whole offscreen texture every frame while panning because for performance reasons. My idea was to implement a tile based progressive rendering technique. Before trying to go this way I would look into these 2 options: make your offscreen texture bigger, so that it is less blurry when zooming in (I guess you don't need very high zoom levels?). Pros: very simple to implement. Cons: uses more memory and rendering to that bigger texture will be slower. directly render everything to the framebuffer. I'm skeptical about you having to go through an offscreen texture. Do you really have so much to render that it's impossible to render in 16ms? Did you profile with Instruments/GPU Frame Capture to see what part is slow and whether it can be improved? Is the bottleneck the rendering or scheduling part?

I think you're trying to over-specify it. When building Metal shaders from Xcode 14.2 there is no "-std" or "-mios-version-min". Instead there is just "-target air64-apple-ios16.2" where 16.2 should be replaced by your iOS deployment target. And this will automatically enable the corresponding available Metal language version. You can add this to one of your shaders and see which warnings get generated: #if __METAL_VERSION__ >= 200 #warning Metal 2.0 #endif #if __METAL_VERSION__ >= 240 #warning Metal 2.4 #endif #if __METAL_VERSION__ >= 300 #warning Metal 3.0 #endif In my tries iOS 15 gives up to Metal 2.4 while setting iOS 16 target will also give Metal 3.0. And iOS 13.5 just gives Metal 2.0 (you can maybe get some intermediate versions between 2.0 and 2.4).

blue reference folder As I understand you got this by adding the directory in your project and choosing "Create folder reference" rather than "Create groups". When doing this, the folder is added in the project hierarchy for easy access from Xcode but is not part of any target, not indexed (so only very basic autocompletion) and consequently lacks highlighting. Unless you make your files belong to a target (thus choosing "Create groups" when adding the folder) I guess you'll only have this limited experience.

I tried a similar way with MTKView as I'm more used to it. class ViewController: UIViewController, MTKViewDelegate {     var mtkView: MTKView!          override func viewDidLoad() {         super.viewDidLoad()         mtkView = MTKView(frame: view.bounds, device: MTLCreateSystemDefaultDevice()!)         mtkView.preferredFramesPerSecond = 120         mtkView.autoResizeDrawable = true         mtkView.delegate = self         mtkView.translatesAutoresizingMaskIntoConstraints = false         view.addSubview(mtkView)                  NSLayoutConstraint.activate([             mtkView.leftAnchor.constraint(equalTo: view.leftAnchor),             mtkView.rightAnchor.constraint(equalTo: view.rightAnchor),             mtkView.topAnchor.constraint(equalTo: view.topAnchor),             mtkView.bottomAnchor.constraint(equalTo: view.bottomAnchor),         ])     }          func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) {}          func draw(in view: MTKView) {         let startTime = CACurrentMediaTime()         let drawable = view.currentDrawable         let timeUsed = CACurrentMediaTime() - startTime         if (timeUsed > 0.003) {             print("CAMetalLayer.nextDrawable take much time!! -> \(String(format: "%.2f", timeUsed * 1000)) ms")         }                  guard let drawable else {             print("no drawable available")             return         }         drawable.present()     } } and it gives similar logs. However, looking at the profiling trace I don't think there's an issue at all: You can see with "LCD" line that surface is regularly displayed every 16ms, and with "Time Profiler" line that CPU isn't busy. This shows that getting the current/next drawable is not slow but is just blocking because it's too early. My understanding is that the draw() method (or in your case the onDisplayLink() method) is called "too early". In fact it's not really too early: it's a bit ahead of time to let you perform some work in advance before you actually need the drawable. This is described in more details there: https://developer.apple.com/library/archive/documentation/3DDrawing/Conceptual/MTLBestPracticesGuide/Drawables.html

What do you see when using the debugger on your fragment shader? https://developer.apple.com/documentation/metal/developing_and_debugging_metal_shaders