anyone know how to rotate the CMSampleBuffer to landscape

Question

michelle211 OP

Created Nov ’19

Replies 3

Boosts 0

Views 5.2k

Participants 5

Has anyone figured this out.

It seems replaykit 2 always delevers frames at a fixed resolution of 886 x 1918

I've tried various solutions to rotate the screen but so far none work.

Using the code below crashes in the rotate step

vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))

  let flags = CVPixelBufferLockFlags(rawValue: 0)
  guard kCVReturnSuccess == CVPixelBufferLockBaseAddress(srcPixelBuffer, flags) else {
    return nil
  }
  defer { CVPixelBufferUnlockBaseAddress(srcPixelBuffer, flags) }

  guard let srcData = CVPixelBufferGetBaseAddress(srcPixelBuffer) else {
    print("Error: could not get pixel buffer base address")
    return nil
  }
  let sourceWidth = CVPixelBufferGetWidth(srcPixelBuffer)
  let sourceHeight = CVPixelBufferGetHeight(srcPixelBuffer)
  var destWidth = sourceHeight
  var destHeight = sourceWidth
  var color = UInt8(0)

  if factor % 2 == 0 {
    destWidth = sourceWidth
    destHeight = sourceHeight
  }

  let srcBytesPerRow = CVPixelBufferGetBytesPerRow(srcPixelBuffer)
  var srcBuffer = vImage_Buffer(data: srcData,
                                height: vImagePixelCount(sourceHeight),
                                width: vImagePixelCount(sourceWidth),
                                rowBytes: srcBytesPerRow)

  let destBytesPerRow = destWidth*4
  guard let destData = malloc(destHeight*destBytesPerRow) else {
    print("Error: out of memory")
    return nil
  }
  var destBuffer = vImage_Buffer(data: destData,
                                 height: vImagePixelCount(destHeight),
                                 width: vImagePixelCount(destWidth),
                                 rowBytes: destBytesPerRow)

  let error = vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))
  if error != kvImageNoError {
    print("Error:", error)
    free(destData)

Boost

Answer 1

coderchou OP

Jan ’20

The following code works, You might have a try. Good luck!

///Rotate CMSampleBufferRef to landscape

- (void)dealWithSampleBuffer:(CMSampleBufferRef)buffer {

CVPixelBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(buffer);

CIImage *ciimage = [CIImage imageWithCVPixelBuffer:pixelBuffer];

size_t width = CVPixelBufferGetWidth(pixelBuffer);

size_t height = CVPixelBufferGetHeight(pixelBuffer);

CFStringRef RPVideoSampleOrientationKeyRef = (__bridge CFStringRef)RPVideoSampleOrientationKey;

NSNumber *orientation = (NSNumber *)CMGetAttachment(buffer, RPVideoSampleOrientationKeyRef,NULL);

NSLog(@"info:%@", orientation);

CGImagePropertyOrientation cgOrientation = kCGImagePropertyOrientationUp;

switch (orientation.unsignedIntValue) {

case kCGImagePropertyOrientationUp:

cgOrientation = kCGImagePropertyOrientationUp;

self.landscape = NO;

break;

case kCGImagePropertyOrientationRight:

cgOrientation = kCGImagePropertyOrientationLeft;

width = CVPixelBufferGetHeight(pixelBuffer);

height = CVPixelBufferGetWidth(pixelBuffer);

self.landscape = YES;

break;

case kCGImagePropertyOrientationLeft:

cgOrientation = kCGImagePropertyOrientationRight;

width = CVPixelBufferGetHeight(pixelBuffer);

height = CVPixelBufferGetWidth(pixelBuffer);

self.landscape = YES;

break;

default:

break;

}

if (self.lastOritation != orientation.unsignedIntValue) {

//init ************ again, Adjust the width and height of the encoding

// [self EndVideoToolBox];

// [self initVideoToolBox];

NSLog(@"orientation changed");

self.lastOritation = orientation.unsignedIntValue;

usleep(5*1000);

return;

}

//Portrait, no need to rotate

if (orientation.unsignedIntegerValue == kCGImagePropertyOrientationUp) {

//encode buffer by ************

// [self encode:buffer];

return ;

}

// roate ciimage

CIImage *wImage = [ciimage imageByApplyingCGOrientation:cgOrientation];

CIImage *newImage = [wImage imageByApplyingTransform:CGAffineTransformMakeScale(0.5, 0.5)];

CVPixelBufferLockBaseAddress(pixelBuffer, 0);

NSLog(@"sample width :%ld height :%ld",width,height)

float scale = 1.0;

CVPixelBufferRef newPixcelBuffer = nil;

CVPixelBufferCreate(kCFAllocatorDefault, width * 0.5 * scale, height * 0.5 * scale, kCVPixelFormatType_32BGRA, nil, &newPixcelBuffer);

if(!_cicontext) {

_cicontext = [CIContext context];

}

[_cicontext render:newImage toCVPixelBuffer:newPixcelBuffer];

CVPixelBufferUnlockBaseAddress(pixelBuffer, 0);

CMVideoFormatDescriptionRef videoInfo = nil;

CMVideoFormatDescriptionCreateForImageBuffer(kCFAllocatorDefault, newPixcelBuffer, &videoInfo);

CMTime duration = CMSampleBufferGetDuration(buffer);

CMTime presentationTimeStamp = CMSampleBufferGetPresentationTimeStamp(buffer);

CMTime decodeTimeStamp = CMSampleBufferGetDecodeTimeStamp(buffer);

CMSampleTimingInfo sampleTimingInfo;

sampleTimingInfo.duration = duration;

sampleTimingInfo.presentationTimeStamp = presentationTimeStamp;

sampleTimingInfo.decodeTimeStamp = decodeTimeStamp;

//

CMSampleBufferRef newSampleBuffer = nil;

CMSampleBufferCreateForImageBuffer(kCFAllocatorMalloc, newPixcelBuffer, true, nil, nil, videoInfo, &sampleTimingInfo, &newSampleBuffer);

//get a rotated CMSampleBufferRef with newSampleBuffer

//encode newSampleBuffer by ************

// [self encode:newSampleBuffer];

// release

CVPixelBufferRelease(newPixcelBuffer);

CFRelease(newSampleBuffer);

}

0

Answer 2

rhythmicfistman OP

Jan ’20

I have problems with Out of Memory kills using this CoreImage approach. And rotating using vImage has inconsistency problems (parts of the image are from future frames).

0

Answer 3

Ilya_Lyaschenko OP

Nov ’20

i faced a such issue and i made this:

create CIContext in SampleHandler.init()

Code Block      if let metalDevice = MTLCreateSystemDefaultDevice() {
      coreImageContext = CIContext(mtlDevice: metalDevice)
    } else {
      coreImageContext = CIContext(options: nil)
    }

2. add rotate method

Code Block private func rotate(_ sampleBuffer: CMSampleBuffer) -> CVPixelBuffer? {
    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else {
      return nil
    }
    var newPixelBuffer: CVPixelBuffer?
    let error = CVPixelBufferCreate(kCFAllocatorDefault,
                    CVPixelBufferGetHeight(pixelBuffer),
                    CVPixelBufferGetWidth(pixelBuffer),
                    kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
                    nil,
                    &newPixelBuffer)
    guard error == kCVReturnSuccess,
       let buffer = newPixelBuffer else {
      return nil
    }
    let ciImage = CIImage(cvPixelBuffer: pixelBuffer).oriented(.left)
    coreImageContext.render(ciImage, to: buffer)
    return buffer
  }

3. use the rotate() method in processSampleBuffer and send rotated buffer to a streaming-service

0