I have created a 3D model of a local outdoor performance space and I have an app that uses
Metal on MacOS 10.14.2 to display the model. I want to create an animation
by flying the camera around the scene, while record each frame. I know how to do
the animated fly-around and I know how to create a video frame-by-frame with
AVFoundation. The step for which I can find no information is how I can capture each frame.
I have a completion handler so I know when the gpu has finished each command buffer.
But what is the best way to get the image in space?
I thought perhaps I could do this by attaching a second texture to colorAttachments[1]
but this has resulted in some odd behavior where the original scene that used to fill my
MTKView window now occupies just the upper left quadrant of the window.
What I was trying to do is write the same color to both colorAttachments[0] (my screen) and to
colorAttachments[1]. In my shader I defined:
struct FragmentOut {
float4 color0 [[ color(0) ]];
float4 color1 [[ color(1) ]];
};
My fragment shader looks like:
fragment FragmentOut grove_fragment_function(VertexOut vIn [[ stage_in ]],
constant Uniforms &uniforms [[buffer(1)]]) {
....
float4 color = diffuseColor + ambientColor + specularColor;
out.color0 = color;
out.color1 = color;
return out;
}
My hope was that I could then use something like:
offScreenTextureBuffer?.getBytes(buffer, bytesPerRow: 4 * w, from: MTLRegionMake2D(0, 0, w, h), mipmapLevel: 0)
to transfer the image data to a local buffer.
This doesn't seem to work, plus I have the unexpected display behavior noted above.
I am configuring the offScreenTextureBuffer thus:
let pixelFormat = MTLPixelFormat.bgra8Unorm_srgb
var offScreenBufferDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: pixelFormat,
width: 1000,
height: 600,
mipmapped: false)
offScreenBufferDescriptor.usage = [.renderTarget, .shaderRead]
offScreenTextureBuffer = device.makeTexture(descriptor: offScreenBufferDescriptor)
I've seen no reply to this question. But after lots of experimentation and further study of Apple documentation for clues I have figured out how to capture GPU generated frames.
First, I gave up on the attempt to have the shader write to the screen and an offscreen buffer in the same render pass. The odd (and undocumented as far as I can tell) behavior of MTKView in which the onscreen view occupies only a portion of the original area of the screen. I think this a bug in MTKView. It seems to act as if each attachment should appear somewhere on the screen. MTKView is not checking to see if the attachments are all on the screen.
Since MTKView calls the draw() function 60 times per second I decided to alternate. One draw() goes to the screen and the following draw() goes to my off screen buffer. This gives me a 30 Hz frame rate for the captured video frames, which is what I wanted anyway.
For the video capture draw calls I now define the MTLTextureDescriptor in this way:
let offScreenPixelFormat = MTLPixelFormat.rgba32Float // 32Float is convenient for AVFoundation video recording
let offScreenBufferDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: offScreenPixelFormat,
width: offW,
height: offH,
mipmapped: false)
offScreenBufferDescriptor.usage = [.shaderWrite, .renderTarget]
offScreenBufferDescriptor.storageMode = .managed
offScreenBufferDescriptor.resourceOptions = [.storageModeManaged]
offScreenBufferDescriptor.textureType = MTLTextureType.type2D
offScreenTextureBuffer = device.makeTexture(descriptor: offScreenBufferDescriptor)It turns out this still did not work. I got nothing back. That is, I got nothing back until I added this code to the end of the commandBuffer:
let blitCommandEncoder = commandBuffer.makeBlitCommandEncoder()
blitCommandEncoder?.synchronize(resource: offScreenTextureBuffer!)
blitCommandEncoder?.endEncoding()This causes the GPU memory version of the offScreenTextureBuffer to be copied back to the CPU memory version. I figured this out by looking at the Apple documentation. Like all Apple documentation, it doesn't tell you why you might want to synchronize. But you definitely do need to do this. I have seen several online code examples where this is NOT done and they claim to get data back from the GPU. So I assume the requirement to synchronize is a new feature and the online code is just out of date. It would be nice if the Apple documentation made it clear this is now a requirement.
With these changes I can retrieve GPU frames.
If anyone is interesting in a complete sample of code, let me know.