Setting a view's CALayer.contents to CVPixelBuffer directly

Question

Created Feb ’21

Replies 4

Boosts 0

Views 3.1k

Participants 2

I'm working on an app in which I create a CVPixelBuffer that I need to display as efficiently as possible. The fastest thing that seems work on a couple recent iPhones I've tested is to simply set the ViewController's layer.contents to the CVPixelBuffer. I was actually surprised this works as googling it would indicate that you can't just do this. However... upon testing this same piece of code on an old iPad Air 2, it didn't display anything. I did succeed in making it work on the iPad Air 2 by creating a CIImage from the CVPixelBuffer and then turning that into a CGImage and setting that to the layer's contents. I'm reluctant to use this in the application as it's considerably less efficient on the iPhone (I'm assuming that this entails some unnecessary extra uploads/downloads to the GPU). I'm trying to figure out which bit of code works on which hardware so I can fall back to the more CPU-intensive function for older hardware, but I'm not sure how to determine which hardware supports setting that CALayer contents to a CVPixelBuffer and which does not. Is there any documentation that would shed light on this?
Thanks!
Michael

Boost

Answer 1

mji83 OP

Feb ’21

After more investigation I think the reason it's working is that my CVPixelBuffer is IOSurface backed. I guess the question is why does setting a CALayer contents to an IOSurface work on some iOS devices and not on others?

0

Answer 2

mji83 OP

Feb ’21

And after further poking around, what I've realized is that this is dependent upon the underlying format of the CVPixelBuffer. Setting CALayer's contents to a CVPixelBuffer actually does work on the iPad I've been testing with, as long as the CVPixelBuffer is in an RGB format. I had been using kCVPixelFormatType_422YpCbCr8 as this is a video streaming application. Interestingly, either format works fine on newer iOS devices. The best solution I have right now is have the app create test buffers, set them to the CALayer contents and then render the CALayer back to a CGImage, sample the CGImage's pixels to test which formats work and which don't. Not ideal, but I can't see any way of determining the pixel format capabilities.

michael

0

Answer 3

ryg OP

Apr ’21

Any sample code you could provide here :)? Very similar situation and my layer with IOSurface contents never updates never updates despite kCVPixelFormatType_32BGRA

0

Answer 4

mji83 OP

Apr ’21

Sure, create a blank project and make this your viewcontroller code. If it works, you'll see a rainbow gradient on your screen. I can verify this works on my iPhone XS

Code Block //
// ViewController.swift
// TestBuffer
//
// Created by Ilardi, Michael on 4/9/21.
//
import UIKit
class ViewController: UIViewController {
   
   
  override func viewDidLoad() {
    super.viewDidLoad()
     
    
    //create a rainbow RGB in a Uint32 array
    let width = 256,height=256
     
    var buffer: [UInt32] = []
     
     
    for i in 0...width - 1 {
      for j in 0...height - 1{
                
        var color:UInt32
        let r = UInt32 ((Float(i)/(Float(height))) * 255.0)
        let g = UInt32 ((Float(j)/(Float(width))) * 255.0)
        let b = UInt32 (128)
        color = (r << 24) + (g << 16) + (b << 8) + 255;
         
        buffer.append(color)
         
      }
    }
     
     
   //now create a CVPixelBuffer and copy our rainbow array into it
     
    let format = kCVPixelFormatType_32ARGB
    
    
    // kCVPixelFormatType_32ARGB
    let sourcePixelBufferOptions: NSDictionary = [kCVPixelBufferPixelFormatTypeKey: format,
      kCVPixelBufferWidthKey: width,
      kCVPixelBufferHeightKey: height,
      kCVPixelBufferIOSurfacePropertiesKey: NSDictionary(),
      kCVPixelBufferIOSurfaceCoreAnimationCompatibilityKey:true,
      kCVPixelBufferCGBitmapContextCompatibilityKey:true
    ]
     
    var pix:CVPixelBuffer?
     
     CVPixelBufferCreate(kCFAllocatorDefault, width, height, format, sourcePixelBufferOptions, &pix)
     
    let flags = CVPixelBufferLockFlags(rawValue: 0)
    CVPixelBufferLockBaseAddress(pix! ,flags)
     
     let pixelBufferAddress = CVPixelBufferGetBaseAddress((pix)!)
   
     
    memcpy(pixelBufferAddress, buffer, 4 * height * width)
    CVPixelBufferUnlockBaseAddress(pix!, flags)
     
     
     
    //set the main view's layer contents to our pixelbuffer
     
    view.layer.contents = pix
     
   
  }
  
}

0