I am trying get WKWebView into a MTLTexture for the purpose of sending pixel data of the web view to an encoder of a live streaming service. After spending some time with the documentation, I have come up with three ideas for loading WKWebView into a MTLTexture.
Idea 1
My first idea is to utilize WKWebView’s `takeSnapshot` function, which returns a snapshot in the form of a `UIImage`. Specifically, I was thinking that I could call `takeSnapshot` on a timer that is synchronized to the display’s refresh rate, access the `cgImage` property of the returned `UIImage`, and pass the `CGImage` to the `newTexture` function of an `MTKTextureLoader` to acquire a `MTLTexture` containing the pixels of the web view. However, this approach does not work because the texture loader fails to load the `CGImage` into a `MTLTexture` (as far as I can tell no error is logged). Moreover, I am concerned that `takeSnapshot` may not be performant enough for my purposes. Does anyone know how to solve the issue that arises when trying to create a `MTLTexture` from the `CGImage` and what are your general thoughts of this approach?
You cand find my implementation of this here: https://paste.ofcode.org/AaF9XfCiXumX4hikBiPCjp
Idea 2
My second idea is to leverage the `drawHierarchy` function of the `UIView` class from which `WKWebView` inherits. The implementation of this approach is equivalent to the implementation of the former except that `takeSnapshot` in the `handleDisplayRefresh` function is replaced by a call to `drawHierarchy`:
In this case, the texture loader managed to create a `MTLTexture` from the `CGImage`, but the apps CPU usage fluctuates between 40-70% on an iPhone 6s.
You can find my implementation of this idea here: https://paste.ofcode.org/ZYfy25hAAs9zfnBXjrXBCS
Idea 3
My final idea was to subclass WKWebView and override its inherited `layerClass` property to return a `CAMetalLayer`. I then thought that I could invoke `CAMetalLayer.nextDrawable()` to get a CAMetalDrawable object that held my desired MTLTexture in its `texture` property. Of course, at closer inspection I realized that the MTLTexture of the returned drawable does not contain the pixel data of the web view, but rather is a blank slate onto which one can draw what one wants to appear on the `CAMetalLayer`. Nevertheless, in the documentation for the `CAMetalLayer` class it says:
"A CAMetalLayer creates a pool of Metal drawable objects (CAMetalDrawable). At any given time, one of these drawable objects contains the contents of the layer."
To me this indicates that there exists a drawable whose texture property indeed contains the contents of the web view. Alas, I do not see any publicly available API to access it. Does anyone know of a technique that would allow me to access the drawable that contains the web view’s pixels?
Lastly, any other ideas for how I could get the pixel data of WKWebView into a MTLTexture (or, otherwise, just the raw pixel data) would be greatly appreciated.
