How to update destinationBuffer's width and height during a video buffer stream

Question

Created Feb ’22

Replies 3

Boosts 0

Views 1k

Participants 2

I am using the Accelerate Framework to convert YUV Data to ARGB Data for a Video Call App. The framework works great, However when I hold calls I use a place holder image sent from the server. That image causes issues sometimes because of its size. Accelerate is telling me that it's range of interest is larger than the input buffer(roiLargerThanInputBuffer). I am not sure exactly how to address this issue. Any thoughts or suggestions would be greatly appreciated.

The problem was that my video buffer stream's pixel buffer width and height changed from the server side. That being said all that needed to be done is to check for when it changes and then remove the current vImage_buffer from memory and reinitialize a new one with the correct size.

Is it proper to tell the accelerate framework to change the vImage_buffer width and height this way. It seems to work well.

 if myBuffer.height != destinationBuffer.height {
            free(destinationBuffer.data)
                error = vImageBuffer_Init(&destinationBuffer,
                                          vImagePixelCount(myBuffer.height),
                                          vImagePixelCount(myBuffer.height),
                                          cgImageFormat.bitsPerPixel,
                                          vImage_Flags(kvImageNoFlags))
                guard error == kvImageNoError else {
                    return nil
                }
        }

Thanks

Boost

Answer 1

Engineer OP

Apple

Feb ’22

Hi,

You can create an intermediate destination buffer that shares your destination's data and rowBytes, but has the size of the source buffer. For example, the following code converts a 4-channel source to a 3-channel destination. The destination is given an arbitrary size.

If your passed the 1500x1500 buffer to vImageConvert_ARGB8888toRGB888, the function would return kvImageRoiLargerThanInputBuffer. Passing fauxBuffer populates the top left corner of destBuffer with the conversion result.

The vImage_Buffer(data: initializer doesn't copy data: fauxBuffer.data points to destBuffer.data.

Although my example is CGImage based, you can use the same technique for your conversion workflow.

/// 8 bit per channel, ARGB

var srcBuffer = try vImage_Buffer(cgImage: cgImage) // `vImageBuffer_InitWithCGImage` with the C API



/// 8 bit per channel, RGB

var destBuffer = vImage_Buffer()

vImageBuffer_Init(&destBuffer,

                  1500, 1500,

                  3 * 8,

                  vImage_Flags(kvImageNoFlags))



var fauxBuffer = vImage_Buffer(data: destBuffer.data,

                               height: srcBuffer.height,

                               width: srcBuffer.width,

                               rowBytes: destBuffer.rowBytes)



let error = vImageConvert_ARGB8888toRGB888(&srcBuffer,

                                           &fauxBuffer,

                                           vImage_Flags(kvImageNoFlags))

On return, destBuffer contains a 3-channel representation of the original CGImage.

1

Answer 2

Engineer OP

Apple

Feb ’22

Hi,

Sorry if my answer wasn't clear. My suggestion is that you call vImageBuffer_Init once with a size that's large enough to handle the different cases. You can create the fauxBuffer with very little cost with each new incoming frame because it doesn't need its data to be allocated. Your original solution allocates memory with vImageBuffer_Init for each size difference between frames and that has a computational cost that the fauxBuffer solution doesn't have.

With my suggestion, you'll need to deallocate destBuffer once your app is finished.

Maybe this makes it clearer:

var destBuffer = vImage_Buffer()

vImageBuffer_Init(&destBuffer,
                  1500, 1500,
                  3 * 8,
                  vImage_Flags(kvImageNoFlags))

while  [continuous data stream is active] {
    let srcBuffer: vImage_Buffer = // derived from continuous data stream

    processFrame(srcBuffer: srcBuffer,
                 destBuffer: destBuffer)
}

destBuffer.data.deallocate()

func processFrame(srcBuffer: vImage_Buffer, destBuffer: vImage_Buffer) {

    var fauxBuffer = vImage_Buffer(data: destBuffer.data,
                                   height: srcBuffer.height,
                                   width: srcBuffer.width,
                                   rowBytes: destBuffer.rowBytes)

    vImageConvert_ARGB8888toRGB888(&srcBuffer,
                                   &fauxBuffer,
                                   vImage_Flags(kvImageNoFlags))
}

0

Answer 3

Cartisim OP

Feb ’22

Ah ok I see. Hey thanks a bunch. The memory footprint is certainly better. Much appreciated!

Question:

Is free(destinationBuffer.data) preferred over destinationBuffer.data.deallocate()?

1