Post

Replies

Boosts

Views

Activity

App using MetalKit creates many IOSurfaces in rapid succession, causing MTKView to freeze and app to hang
I've got an iOS app that is using MetalKit to display raw video frames coming in from a network source. I read the pixel data in the packets into a single MTLTexture rows at a time, which is drawn into an MTKView each time a frame has been completely sent over the network. The app works, but only for several seconds (a seemingly random duration), before the MTKView seemingly freezes (while packets are still being received). Watching the debugger while my app was running revealed that the freezing of the display happened when there was a large spike in memory. Seeing the memory profile in Instruments revealed that the spike was related to a rapid creation of many IOSurfaces and IOAccelerators. Profiling CPU Usage shows that CAMetalLayerPrivateNextDrawableLocked is what happens during this rapid creation of surfaces. What does this function do? Being a complete newbie to iOS programming as a whole, I wonder if this issue comes from a misuse of the MetalKit library. Below is the code that I'm using to render the video frames themselves: class MTKViewController: UIViewController, MTKViewDelegate { /// Metal texture to be drawn whenever the view controller is asked to render its view. private var metalView: MTKView! private var device = MTLCreateSystemDefaultDevice() private var commandQueue: MTLCommandQueue? private var renderPipelineState: MTLRenderPipelineState? private var texture: MTLTexture? private var networkListener: NetworkListener! private var textureGenerator: TextureGenerator! override public func loadView() { super.loadView() assert(device != nil, "Failed creating a default system Metal device. Please, make sure Metal is available on your hardware.") initializeMetalView() initializeRenderPipelineState() networkListener = NetworkListener() textureGenerator = TextureGenerator(width: streamWidth, height: streamHeight, bytesPerPixel: 4, rowsPerPacket: 8, device: device!) networkListener.start(port: NWEndpoint.Port(8080)) networkListener.dataRecievedCallback = { data in self.textureGenerator.process(data: data) } textureGenerator.onTextureBuiltCallback = { texture in self.texture = texture self.draw(in: self.metalView) } commandQueue = device?.makeCommandQueue() } public func mtkView(_ view: MTKView, drawableSizeWillChange size: CGSize) { /// need implement? } public func draw(in view: MTKView) { guard let texture = texture, let _ = device else { return } let commandBuffer = commandQueue!.makeCommandBuffer()! guard let currentRenderPassDescriptor = metalView.currentRenderPassDescriptor, let currentDrawable = metalView.currentDrawable, let renderPipelineState = renderPipelineState else { return } currentRenderPassDescriptor.renderTargetWidth = streamWidth currentRenderPassDescriptor.renderTargetHeight = streamHeight let encoder = commandBuffer.makeRenderCommandEncoder(descriptor: currentRenderPassDescriptor)! encoder.pushDebugGroup("RenderFrame") encoder.setRenderPipelineState(renderPipelineState) encoder.setFragmentTexture(texture, index: 0) encoder.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, instanceCount: 1) encoder.popDebugGroup() encoder.endEncoding() commandBuffer.present(currentDrawable) commandBuffer.commit() } private func initializeMetalView() { metalView = MTKView(frame: CGRect(x: 0, y: 0, width: streamWidth, height: streamWidth), device: device) metalView.delegate = self metalView.framebufferOnly = true metalView.colorPixelFormat = .bgra8Unorm metalView.contentScaleFactor = UIScreen.main.scale metalView.autoresizingMask = [.flexibleWidth, .flexibleHeight] view.insertSubview(metalView, at: 0) } /// initializes render pipeline state with a default vertex function mapping texture to the view's frame and a simple fragment function returning texture pixel's value. private func initializeRenderPipelineState() { guard let device = device, let library = device.makeDefaultLibrary() else { return } let pipelineDescriptor = MTLRenderPipelineDescriptor() pipelineDescriptor.rasterSampleCount = 1 pipelineDescriptor.colorAttachments[0].pixelFormat = .bgra8Unorm pipelineDescriptor.depthAttachmentPixelFormat = .invalid /// Vertex function to map the texture to the view controller's view pipelineDescriptor.vertexFunction = library.makeFunction(name: "mapTexture") /// Fragment function to display texture's pixels in the area bounded by vertices of `mapTexture` shader pipelineDescriptor.fragmentFunction = library.makeFunction(name: "displayTexture") do { renderPipelineState = try device.makeRenderPipelineState(descriptor: pipelineDescriptor) } catch { assertionFailure("Failed creating a render state pipeline. Can't render the texture without one.") return } } } My question is simply: what gives?
1
0
665
Jun ’24