anyone know how to rotate the CMSampleBuffer to landscape

Has anyone figured this out.


It seems replaykit 2 always delevers frames at a fixed resolution of 886 x 1918


I've tried various solutions to rotate the screen but so far none work.


Using the code below crashes in the rotate step


vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))


  let flags = CVPixelBufferLockFlags(rawValue: 0)
  guard kCVReturnSuccess == CVPixelBufferLockBaseAddress(srcPixelBuffer, flags) else {
    return nil
  }
  defer { CVPixelBufferUnlockBaseAddress(srcPixelBuffer, flags) }

  guard let srcData = CVPixelBufferGetBaseAddress(srcPixelBuffer) else {
    print("Error: could not get pixel buffer base address")
    return nil
  }
  let sourceWidth = CVPixelBufferGetWidth(srcPixelBuffer)
  let sourceHeight = CVPixelBufferGetHeight(srcPixelBuffer)
  var destWidth = sourceHeight
  var destHeight = sourceWidth
  var color = UInt8(0)

  if factor % 2 == 0 {
    destWidth = sourceWidth
    destHeight = sourceHeight
  }

  let srcBytesPerRow = CVPixelBufferGetBytesPerRow(srcPixelBuffer)
  var srcBuffer = vImage_Buffer(data: srcData,
                                height: vImagePixelCount(sourceHeight),
                                width: vImagePixelCount(sourceWidth),
                                rowBytes: srcBytesPerRow)

  let destBytesPerRow = destWidth*4
  guard let destData = malloc(destHeight*destBytesPerRow) else {
    print("Error: out of memory")
    return nil
  }
  var destBuffer = vImage_Buffer(data: destData,
                                 height: vImagePixelCount(destHeight),
                                 width: vImagePixelCount(destWidth),
                                 rowBytes: destBytesPerRow)

  let error = vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))
  if error != kvImageNoError {
    print("Error:", error)
    free(destData)

Replies

The following code works, You might have a try. Good luck!


///Rotate CMSampleBufferRef to landscape

- (void)dealWithSampleBuffer:(CMSampleBufferRef)buffer {

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(buffer);


CIImage *ciimage = [CIImage imageWithCVPixelBuffer:pixelBuffer];


size_t width = CVPixelBufferGetWidth(pixelBuffer);

size_t height = CVPixelBufferGetHeight(pixelBuffer);



CFStringRef RPVideoSampleOrientationKeyRef = (__bridge CFStringRef)RPVideoSampleOrientationKey;

NSNumber *orientation = (NSNumber *)CMGetAttachment(buffer, RPVideoSampleOrientationKeyRef,NULL);

NSLog(@"info:%@", orientation);


CGImagePropertyOrientation cgOrientation = kCGImagePropertyOrientationUp;


switch (orientation.unsignedIntValue) {

case kCGImagePropertyOrientationUp:

cgOrientation = kCGImagePropertyOrientationUp;

self.landscape = NO;

break;

case kCGImagePropertyOrientationRight:

cgOrientation = kCGImagePropertyOrientationLeft;

width = CVPixelBufferGetHeight(pixelBuffer);

height = CVPixelBufferGetWidth(pixelBuffer);

self.landscape = YES;

break;

case kCGImagePropertyOrientationLeft:

cgOrientation = kCGImagePropertyOrientationRight;

width = CVPixelBufferGetHeight(pixelBuffer);

height = CVPixelBufferGetWidth(pixelBuffer);

self.landscape = YES;

break;

default:

break;

}



if (self.lastOritation != orientation.unsignedIntValue) {

//init ************ again, Adjust the width and height of the encoding

// [self EndVideoToolBox];

// [self initVideoToolBox];

NSLog(@"orientation changed");

self.lastOritation = orientation.unsignedIntValue;

usleep(5*1000);

return;

}



//Portrait, no need to rotate

if (orientation.unsignedIntegerValue == kCGImagePropertyOrientationUp) {

//encode buffer by ************

// [self encode:buffer];

return ;

}



// roate ciimage

CIImage *wImage = [ciimage imageByApplyingCGOrientation:cgOrientation];


CIImage *newImage = [wImage imageByApplyingTransform:CGAffineTransformMakeScale(0.5, 0.5)];

CVPixelBufferLockBaseAddress(pixelBuffer, 0);



NSLog(@"sample width :%ld height :%ld",width,height)


float scale = 1.0;

CVPixelBufferRef newPixcelBuffer = nil;

CVPixelBufferCreate(kCFAllocatorDefault, width * 0.5 * scale, height * 0.5 * scale, kCVPixelFormatType_32BGRA, nil, &newPixcelBuffer);



if(!_cicontext) {

_cicontext = [CIContext context];

}

[_cicontext render:newImage toCVPixelBuffer:newPixcelBuffer];



CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);

CMVideoFormatDescriptionRef videoInfo = nil;

CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, newPixcelBuffer, &videoInfo);

CMTime duration = CMSampleBufferGetDuration(buffer);

CMTime presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(buffer);

CMTime decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(buffer);

CMSampleTimingInfo sampleTimingInfo;

sampleTimingInfo.duration = duration;

sampleTimingInfo.presentationTimeStamp = presentationTimeStamp;

sampleTimingInfo.decodeTimeStamp = decodeTimeStamp;

//

CMSampleBufferRef newSampleBuffer = nil;

CMSampleBufferCreateForImageBuffer(kCFAllocatorMalloc, newPixcelBuffer, true, nil, nil, videoInfo, &sampleTimingInfo, &newSampleBuffer);


//get a rotated CMSampleBufferRef with newSampleBuffer


//encode newSampleBuffer by ************

// [self encode:newSampleBuffer];



// release

CVPixelBufferRelease(newPixcelBuffer);

CFRelease(newSampleBuffer);

}

I have problems with Out of Memory kills using this CoreImage approach. And rotating using vImage has inconsistency problems (parts of the image are from future frames).

i faced a such issue and i made this:
  1. create CIContext in SampleHandler.init()

Code Block
     if let metalDevice = MTLCreateSystemDefaultDevice() {
      coreImageContext = CIContext(mtlDevice: metalDevice)
    } else {
      coreImageContext = CIContext(options: nil)
    }


2. add rotate method
Code Block
private func rotate(_ sampleBuffer: CMSampleBuffer) -> CVPixelBuffer? {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
      return nil
    }
    var newPixelBuffer: CVPixelBuffer?
    let error = CVPixelBufferCreate(kCFAllocatorDefault,
                    CVPixelBufferGetHeight(pixelBuffer),
                    CVPixelBufferGetWidth(pixelBuffer),
                    kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
                    nil,
                    &newPixelBuffer)
    guard error == kCVReturnSuccess,
       let buffer = newPixelBuffer else {
      return nil
    }
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer).oriented(.left)
    coreImageContext.render(ciImage, to: buffer)
    return buffer
  }


3. use the rotate() method in processSampleBuffer and send rotated buffer to a streaming-service
  • your code will make crash due to lack memory

Add a Comment