WKWebView offscreen rendering

Question

Joey_Yu OP

Created Jul ’22

Replies 3

Boosts 0

Views 2.0k

Participants 2

I'm trying to integrate WKWebView to metal rendering pipline and met 2 issues.

WKWebView will kick webview rendering only when the WKWebView control is visible, i have been looking around WKWebView related header files and configurations, didn't find a way to force WkWebView render the view for every single frame when the control is invisible or not part of view hierarchy .
Low performance when access CGImage pixel data returned from WKWebview takesnapshot, in order to get the pixel data uploaded to MTLTexture i did the following tests:

Access the pixel data through CGDataProviderCopyData, the CPU usage is very high, main thread drop to 37 FPS on IPhone 8 Plus simulator, most CPU cycles spend on function vConvert_PermuteChannels_ARGB8888_CV_vec, i also try to render the CGImage to a BitmapContext but still can't get ride of vConvert_PermuteChannels_ARGB8888_CV_vec.

      CGDataProviderRef provider = CGImageGetDataProvider(cgImage);
      CFDataRef rawData = CGDataProviderCopyData( provider );
      CFIndex length = CFDataGetLength( rawData );
      UInt8 *buf = (UInt8*) CFDataGetBytePtr( rawData );
      MTLRegion region = {
        {0, 0, 0},
        {1280, 960, 1}
      };
   
      [_webviewTexture replaceRegion:region mipmapLevel:0 withBytes:buf bytesPerRow:bytesPerRow];
      CFRelease( rawData );

Another try is create metal texture from CGImage via textureLoader but failed with error "image decoding failed"

       MTKTextureLoader *loader = [[MTKTextureLoader alloc] initWithDevice: _device];
      NSDictionary *textureLoaderOption = @{
        MTKTextureLoaderOptionTextureUsage:@(MTLTextureUsageShaderRead),
        MTKTextureLoaderOptionTextureStorageMode : @(MTLStorageModePrivate)
      };
      NSError *error = nil;
      _webviewTexture = [ loader newTextureWithCGImage:cgImage options:textureLoaderOption error:&error];

Any idea against how to force kick WebView rendering or more efficient way to access CGImage raw pixel data would be greatly appreciated.

Boost

Answer 1

Joey_Yu OP

Jul ’22

Regarding upload CGImage pixel data to MTLTexture, i also tried vImage acclerated API, the performance bottleneck still lies on vConvert_PermuteChannels_ARGB8888_CV_vec, considering the function name start with "v", i guess it's already SIMD like instruction set accelerated. and channel permute operation looks like can't be avoided.

0

Answer 2

Ceylo OP

Jul ’22

I'm not sure everything will help, and I don't know for the WebKit offscreen rendering, but here are at least three points I can mention:

Don't take iPhone simulator as a reference for your benchmark, use a real device
Being on iPhone you can take advantage of the unified memory architecture and create textures without doing any copy, if the source data is properly allocated and aligned. In particular see https://developer.apple.com/documentation/metal/mtldevice/1433382-makebuffer and https://developer.apple.com/documentation/metal/mtlbuffer/1613852-maketexture. This means that the CGImage buffers in which you render to must have been allocated by you, following above constraints, and that the CGImage must only wrap your pointers, not copy your data to its own buffers (I'm not sure if CGImage can do that, so you might need to render into something else than a CGImage).
If the size of the texture doesn't change, you can reuse the texture but make sure it's not used by Metal while you write to it: either you wait for MTLCommandBuffer to complete, or you create several buffers/textures that you reuse over time to account for triple buffering of your rendering.

0

Answer 3

Joey_Yu OP

Jul ’22

Here are the answers:

It's not possible for the time being, WKWebView does not expose enough public API to support what i'm trying to do which let WKWebView works like a offscreen component instead of a UI view control.
There is no faster way to access the pixel data from the CGImage.

0