Cropping CVPixelBuffer with Accelerate + vImageScale

I'm employing this technique iOS - Scale and crop CMSampleBufferRef/CVImageBufferRef to crop my sample buffers from a 4:3 aspect ratio to a 16:9 aspect ratio.

But the frames are coming out as all black in my MTKView...


I'm trying to get this cropping done in real time, hence the use of vImage. Definitely open to another real-time method if there is one!


My camera is configured to output

kCVPixelFormatType_32BGRA
pixels, and this problem is on the fourThreeActive = true branch.


When I convert the

pixelBuffer
to a
UIImage
immediately after the vImage operation to check the output.. i get a weird "scrambled" image (nothing is discrenable... looks like noise on a tv). When I convert to a UIImage before the vImage operation... the image looks correct.


-(void)captureOutput:(AVCaptureOutput *)captureOutput didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer fromConnection:(AVCaptureConnection *)connection {
           
       CVImageBufferRef pixelBuffer = NULL;
           
       if (fourThreeActive) {
               
          CVImageBufferRef imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
               
          size_t startingWidth = CVPixelBufferGetWidth(imageBuffer);
          size_t cropInsetX = (int)(((1 - (4.0/3.0)/(16.0/9.0)) * 0.5) * startingWidth);
          size_t finalWidth = startingWidth - cropInsetX * 2;
          size_t height = CVPixelBufferGetHeight(imageBuffer);
               
          CVPixelBufferLockBaseAddress(imageBuffer, 0);
               
          void *baseAddress = CVPixelBufferGetBaseAddress(imageBuffer);
             
          void* destData = malloc(height * finalWidth * 4);
               
          vImage_Buffer srcBuffer = { (void *)baseAddress + cropInsetX * 4, height, finalWidth, finalWidth * 4};
          vImage_Buffer destBuffer = { (void *)destData, height, finalWidth, finalWidth * 4};
               
          vImage_Error err = vImageScale_ARGB8888(&srcBuffer, &destBuffer, NULL, 0);
          CVPixelBufferUnlockBaseAddress(imageBuffer, 0);
               
          if (err != kvImageNoError) {
                   
             NSLog(@"Error: %ld", err);
             free(destData);
          }
               
          OSType pixelFormat = CVPixelBufferGetPixelFormatType(imageBuffer);
            
          CVReturn result = CVPixelBufferCreateWithBytes(NULL, finalWidth, height, pixelFormat, destData, finalWidth * 4, stillImageDataReleaseCallback, NULL, NULL, &pixelBuffer);


          if (result != kCVReturnSuccess) {
                   
             NSLog(@"Error: could not create new pixel buffer");
             free(destData);
          }
       }
       else pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
               
       CVMetalTextureRef texture = NULL;
               
       if (CVMetalTextureCacheCreateTextureFromImage(kCFAllocatorDefault, textureCache, pixelBuffer, NULL, MTLPixelFormatBGRA8Unorm, CVPixelBufferGetWidth(pixelBuffer), CVPixelBufferGetHeight(pixelBuffer), 0, &texture) == kCVReturnSuccess) {
                   
          cameraContainer.metalCamera.workingTexture = CVMetalTextureGetTexture(texture);
          CFRelease(texture);
       }
    }

Accepted Reply

The vimage rowbytes should be the row length of the orignal buffer, not the cropped one.


size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
vImage_Buffer srcBuffer = { (void *)baseAddress + cropInsetX * 4, height, finalWidth, bytesPerRow};

Replies

The vimage rowbytes should be the row length of the orignal buffer, not the cropped one.


size_t bytesPerRow = CVPixelBufferGetBytesPerRow(imageBuffer);
vImage_Buffer srcBuffer = { (void *)baseAddress + cropInsetX * 4, height, finalWidth, bytesPerRow};

I figured this out the other day 🙂 forgot to come back to this. thank you!

Hello, I am trying to achieve the same thing by cropping the CVPixelBuffer, but i am getting a memory leak issue, I am already releasing the CVPixelBufferRef object, is there anything else we need to release ? If possible can you please post stillImageDataReleaseCallback method here?