MTKView fullscreen stutter

hello, when I do Metal drawing into an MTKView in full screen, there's an issue with frame scheduling, it seems. There is visible stutter, and the Metal HUD shows the frame rate jittering about.

happens in Ventura and Sonoma b2 on my Macbook Pro.

here's a really minimal example. Not even actively drawing anything, just presenting the drawable.

#import "ViewController.h"
#import <Metal/Metal.h>
#import <MetalKit/MetalKit.h>

@interface ViewController() <MTKViewDelegate>
@property (nonatomic, weak) IBOutlet MTKView *mtkView;
@property (nonatomic) id<MTLDevice> device;
@property (nonatomic) id<MTLCommandQueue> commandQueue;
@end

@implementation ViewController

- (void)viewDidLoad {
    [super viewDidLoad];

    _device = MTLCreateSystemDefaultDevice();
    _mtkView.device = _device;
    _mtkView.delegate = self;
    _commandQueue = [_device newCommandQueue];
}

- (void)drawInMTKView:(MTKView *)view {
    MTLRenderPassDescriptor *viewRPD = view.currentRenderPassDescriptor;
    if(viewRPD) {
        id<MTLCommandBuffer> commandBuffer = [_commandQueue commandBuffer];
        [commandBuffer presentDrawable:view.currentDrawable];
        [commandBuffer commit];
    }
}


- (void)mtkView:(MTKView *)view drawableSizeWillChange:(CGSize)size {
    NSLog(@"%@", NSStringFromSize(size));
}

Looks like there's some collision between display and render timer, or something. what gives? I would like to be able to render stutter free on this very nice machine? how would I go about that?

Replies

just to be sure, totally freshly nuked macOS 13.3 install and this is definitely still happening.

disabling ProMotion, setting refresh rate to 60Hz can sometimes lead to a functioning-as-expected state:

but it's like rolling dice, other times framerate oscillates so that it looks like it locked to 40 Hz:

a big factor seems to be: are other UI elements composited on top. while hovering the mouse cursor on the top screen edge so the menu bar becomes visible, for example, the stuttering goes away.

I have tried skipping MTKView altogether, using CAMetalLayer and a CVDisplayLink in a NSView subclass, replicating the MTKView functionality. The results are exactly the same.

to recap:

  • if the window is not in fullscreen, the displayLink and presentDrawable work together in lockstep as they should, giving stable 60Hz (or 120Hz).
  • if the window is in fullscreen, there is some massive issue where whatever happens in presentDrawable can't keep up with the displayLink
  • enabling or disabling ProMotion only has an effect on how the issue presents. The issue is present in both modes
  • depending on what is rendered on screen, and how sensitive you are to the resulting stuttering, you may not see the effect with your eyes
  • try rendering a something with sharp edges that moves steadily across the screen, behold the jittering
  • the issue does NOT occur on my olde intel Macbook Air
  • it's easily reproducible and I can't make it go away on my M2 Max Macbook Pro

You have to do your own frame pacing on the present call. See the WWDC 21 presentation all about this. Also the display link fps settings, currentRenderPassDescriptor/currentDrawable calls are all stalls too.

but the frame pacing is solid when not in full screen. it's even solid when in full screen but with the menu bar visible. see this slo-mo video: https://www.youtube.com/watch?v=-YPRlnTc2K0

note how rendering starts fluid with the OS rendering the menu bar, but then starts to wonk out as soon as we are really full screen.

So this is clearly an error.

I'm currently finding that on Ventura, this is borked (will remain borked?).

On Sonoma, it's relatively solid, with the exception that if ProMotion is enabled, and you set MTKView preferredFramesPerSecond to 60, it's very jittery (and with the same feature that if some other UI element is rendered over the fullscreen view, the error subsides).

I am experiencing exactly the same issue under exactly the same conditions in MTKView on MacBook Pro M2 Max. We must be the only 2 people in the world trying to render figures with sharp edges smoothly moving across the screen! We must be doing something wrong, because on the same Mac, the screensaver with smoothly moving curved lines looks beautiful.

I also watched the 2021 WWDC video on frame pacing, and it does help to use present(drawable, afterMinimumDuration: 1.0/FPS), however it again only works when not maximised (or maximised and some controls are hovering on top).

It seems that the system is ignoring afterMinimumDuration when it full screen mode with nothing else visible.

I'm thinking that the only way out might be by computing "by how much the shape should move" based on the time elapsed between two calls to draw(). However, I'm not sure this time will actually correspond to how long this frame will be displayed on the screen later.

Please, please let me know if you find a solution.

This can also be related to how the triple-buffering mechanism of the MTKView works. I think the following might happen. At the beginning of rendering, it has two "next" frames into which is can draw, none of them "in flight" yet. So it calls draw() for the first one, and then immediately after 1/120 second for the second one. Then after 1/120 second more no frames are ready and then after 1/120 both of the frames might be ready, and then frame skip happens, and it calls draw() again on two consecutive steps.

I did the following experiment. Added usleep(20000) at the beginning of a draw() call, and added the Metal monitoring information onto the screen. It shows that the time between frames is sometimes still 1/120, even though it should be impossible, because I put delay of 20 ms into each draw() call. How is that possible? My only explanation is that it might be using multithreaded calls to draw().

Anyway, I'm running out of ideas ;)

My answer doesn't include code, but I believe it will be helpful as I spent 3 days before I figured it out.

You need to use a semaphore to control access to the "in-flight" frames, and the maximum of the "in-flight" frames should be equal to 3 (because MTKView uses triple buffering).

To find out how to do this correctly is easy. Start a new "Game" project in xcode, and select the framework "Metal" (not "SceneKit" or "SpriteKit"), and you can skip tests generation. Then you can find the relevant code in the "Renderer.swift" file (or whatever language you selected).

I commented out all the "cube rotation" rendering in the default project, and just left the drawable presentation. The glitch disappeared at 60 FPS when the window is maximised (or not). Additionally I tried rendering the full video sample with graphics at 120 FPS (by setting preferredFrameRate on the view), and it works marvellously. You can enable metal information display in the options to see that there is no fluctuation if frame rate. Well, there is, but not glitchy anymore.

I'm using MacBook Pro M2.

Finally, Sonoma came out and all the problems went away! I guess I'll have to release my app with minimum MacOS version requirement of 14.0. Can't let the people see what happens on <14...

I'm still seeing this with my app targeting 14.2. I have the semaphore in place as well to prevent stalling when getting the current drawable. I noticed in Instruments that sometimes my present command is failing with a message of "Cannot go direct to display due to display rejected". Any ideas what is going on?

This seems to still be an issue in Sonoma 14.4.1. When in the direct presentation mode, calling presentAfterMinimumDuration with a 16ms present duration consistently results in 25ms presents on a ProMotion display. This directly contradicts documentation here and leaves me with no idea how I am meant to present frames at a regular interval that I specify. CAMetalDisplayLink also seems to have this issue as well as MTKViewDelegate. Is the direct presentation mode just broken? Has anyone filed a feedback?

I've finally figured this out. If anyone else is having trouble with this, verify that the mouse cursor is completely hidden when your application has exclusive fullscreen in the direct present mode. I had very uneven frame pacing with the direct present mode with the cursor onscreen which completely went away after calling hide on the cursor.

I can only speculate what's going on under the hood here. Perhaps it's that on each draw call the system has to check in with the cursor, but with your app having fullscreen priority, the system responds more slowly to requests to render the cursor, causing it to block for a long period before it can actually render the completed drawable for the display?