ARSCNView.snapshot takes 15-19ms: 60fps recording not possible?

I'm trying to take snapshots of my ARSCNView, to put them into a video composition via AVAssetWriterInputPixelBufferAdaptor. This is all for iPhone X only, at the moment.


I've been measuring the time taken for a call to ARSCNView.snapshot(), and it takes 15-19ms with my configuration. That's using the ARKit default video format (it didn't seem to be allowing me to change this, I think it's 1280x720 and at least 60fps). The snapshot yields an image of both the video capture frame and the 3D scene on top. I doubt this matters, but my 3D scene is very simple - one low poly model is in the scene, nothing else.


If this call takes 19ms, we can't snapshot 60 times per second, so there won't be consistent 60fps video created.


Has anyone used the snapshot method in this way, and do you know whether this sounds right? The device is powerful, and I hoped that there'd be a way to record at this rate. I've seen other ways to access SceneKit data, like having a second SCNSceneRenderer, and this has worked for me when recording non-ARKit SceneKit scenes, but I need to be able to obtain the video capture data as well as the 3D scene, and ARKit's snapshot is the only way I've found that gives this. There's a capturedImage property that returns a CVPixelBuffer, but that's only the video capture and not the overlaid 3D scene. Perhaps that means I could take this capturedImage video data, and access the 3D scene via OpenGL, and then combined them myself, but I can't imagine why that approach would be automatically more efficient than an internal method provided by ARKit/SceneKit APIs.


Thanks

Replies

Doing this through snapshots sounds like the wrong approach to me. Have you looked into ReplayKit? It contains functionality for screen capture/recording -- see RPScreenRecorder.

Thanks for your suggestion. Would that not mean the whole app, including the UI, is recorded too? I don't want that - I want to record the relevant AR experience to be used later as media, and that's not great if there are buttons and text top (and those do need to stay on screen, I can't just hide them during recording).

ReplayKit records the main UIWindow of the app. If a second window instance is overlayed with UI controls it will not be recorded.

Doesn't RK require a permissions prompt even if camera permission has already been granted? That's a non-starter in my app due to the added friction.

In order to create a composited video of only the AR scene, you'll need to implement your own SCNRenderer and use the captured image from ARKit along with the Scene from the ARSCN. Theres a great open source framework (ARVideokit) that implements a custom renderer and allows you to create a video recording of the ARSCN along with GIf and LivePhotos.


ARVideoKit: https://github.com/AFathi/ARVideoKit