I am drawing stuff onto an off-screen MTLTexture. (using Skia Canvas)
At a later point, I want to render this MTLTexture into a CAMetalLayer to display it on the screen.
Since I was using Skia for the off-screen drawing operations, my code is quite simple and I don't have the typical Metal setup (no MTLLibrary, MTLRenderPipelineDescriptor, MTLRenderPassDescriptor, MTLRenderEncoder, etc).
I now simply want to draw that MTLTexture into a CAMetalLayer, but haven't figured out how to do so simply.
This is where I draw my stuff to the MTLTexture _texture (Skia code):
- (void) renderNewFrameToCanvas(Frame frame) {
if (_skContext == nullptr) {
GrContextOptions grContextOptions;
_skContext = GrDirectContext::MakeMetal((__bridge void*)_device,
// TODO: Use separate command queue for this context?
(__bridge void*)_commandQueue,
grContextOptions);
}
@autoreleasepool {
// Lock Mutex to block the runLoop from overwriting the _texture
std::lock_guard lockGuard(_textureMutex);
auto texture = _texture;
// Get & Lock the writeable Texture from the Metal Drawable
GrMtlTextureInfo fbInfo;
fbInfo.fTexture.retain((__bridge void*)texture);
GrBackendRenderTarget backendRT(texture.width,
texture.height,
1,
fbInfo);
// Create a Skia Surface from the writable Texture
auto skSurface = SkSurface::MakeFromBackendRenderTarget(_skContext.get(),
backendRT,
kTopLeft_GrSurfaceOrigin,
kBGRA_8888_SkColorType,
nullptr,
nullptr);
auto canvas = skSurface->getCanvas();
auto surface = canvas->getSurface();
// Clear anything that's currently on the Texture
canvas->clear(SkColors::kBlack);
// Converts the Frame to an SkImage - RGB.
auto image = SkImageHelpers::convertFrameToSkImage(_skContext.get(), frame);
canvas->drawImage(image, 0, 0);
// Flush all appended operations on the canvas and commit it to the SkSurface
canvas->flush();
// TODO: Do I need to commit?
/*
id<MTLCommandBuffer> commandBuffer([_commandQueue commandBuffer]);
[commandBuffer commit];
*/
}
}
Now, since I have the MTLTexture _texture in memory, I want to draw it to the CAMetalLayer _layer. This is what I have so far:
- (void) setup {
// I set up a runLoop that calls render() 60 times a second.
// [removed to simplify]
_renderPassDescriptor = [[MTLRenderPassDescriptor alloc] init];
// Load the compiled Metal shader (PassThrough.metal)
auto baseBundle = [NSBundle mainBundle];
auto resourceBundleUrl = [baseBundle URLForResource:@"VisionCamera" withExtension:@"bundle"];
auto resourceBundle = [[NSBundle alloc] initWithURL:resourceBundleUrl];
auto shaderLibraryUrl = [resourceBundle URLForResource:@"PassThrough" withExtension:@"metallib"];
id<MTLLibrary> defaultLibrary = [_device newLibraryWithURL:shaderLibraryUrl error:nil];
id<MTLFunction> vertexFunction = [defaultLibrary newFunctionWithName:@"vertexPassThrough"];
id<MTLFunction> fragmentFunction = [defaultLibrary newFunctionWithName:@"fragmentPassThrough"];
// Create a Pipeline Descriptor that connects the CPU draw operations to the GPU Metal context
auto pipelineDescriptor = [[MTLRenderPipelineDescriptor alloc] init];
pipelineDescriptor.label = @"VisionCamera: Frame Texture -> Layer Pipeline";
pipelineDescriptor.vertexFunction = vertexFunction;
pipelineDescriptor.fragmentFunction = fragmentFunction;
pipelineDescriptor.colorAttachments[0].pixelFormat = MTLPixelFormatBGRA8Unorm;
_pipelineState = [_device newRenderPipelineStateWithDescriptor:pipelineDescriptor error:nil];
}
- (void) render() {
@autoreleasepool {
// Blocks until the next Frame is ready (16ms at 60 FPS)
auto drawable = [_layer nextDrawable];
std::unique_lock lock(_textureMutex);
auto texture = _texture;
MTLRenderPassDescriptor* renderPassDescriptor = [[MTLRenderPassDescriptor alloc] init];
renderPassDescriptor.colorAttachments[0].texture = drawable.texture;
renderPassDescriptor.colorAttachments[0].loadAction = MTLLoadActionClear;
renderPassDescriptor.colorAttachments[0].clearColor = MTLClearColor();
id<MTLCommandBuffer> commandBuffer([_commandQueue commandBuffer]);
auto renderEncoder = [commandBuffer renderCommandEncoderWithDescriptor:renderPassDescriptor];
[renderEncoder setLabel:@"VisionCamera: PreviewView Texture -> Layer"];
[renderEncoder setRenderPipelineState:_pipelineState];
[renderEncoder setFragmentTexture:texture atIndex:0];
[renderEncoder endEncoding];
[commandBuffer presentDrawable:drawable];
[commandBuffer commit];
lock.unlock();
}
}
And along with that, I have created the PassThrough.metal shader which is just for passing through a texture:
#include <metal_stdlib>
using namespace metal;
// Vertex input/output structure for passing results from vertex shader to fragment shader
struct VertexIO
{
float4 position [[position]];
float2 textureCoord [[user(texturecoord)]];
};
// Vertex shader for a textured quad
vertex VertexIO vertexPassThrough(const device packed_float4 *pPosition [[ buffer(0) ]], const device packed_float2 *pTexCoords [[ buffer(1) ]], uint vid [[ vertex_id ]]) {
VertexIO outVertex;
outVertex.position = pPosition[vid];
outVertex.textureCoord = pTexCoords[vid];
return outVertex;
}
// Fragment shader for a textured quad
fragment half4 fragmentPassThrough(VertexIO inputFragment [[ stage_in ]], texture2d<half> inputTexture [[ texture(0) ]], sampler samplr [[ sampler(0) ]]) {
return inputTexture.sample(samplr, inputFragment.textureCoord);
}
Running this crashes the app with the following exception:
validateRenderPassDescriptor:782: failed assertion `RenderPass Descriptor Validation
Texture at colorAttachment[0] has usage (0x01) which doesn't specify MTLTextureUsageRenderTarget (0x04)
This now raises three questions for me:
- Do I have to do all of that Metal setting up, packing along the
PassThrough.metalshader, render pass stuff, etc just to draw theMTLTextureto theCAMetalLayer? Is there no simpler way? - Why is the code above failing?
- When is the drawing from Skia actually committed to the MTLTexture? Do I need to commit the command buffer (as seen in my TODO)?
