Frame orientation when compressing to H264 using Video Toolbox?

Media Technologies Audio
MikeAlpha OP
Created Nov ’17
Replies 6
Boosts 0
Views 3.2k
Participants 4

Hello


I am using ReplayKit to record iOS screen (system-wide recording, started from Control Center) on iPhone 7/iOS 11.1. So far so good, I am getting frames, pushing them into Video Toolbox, then getting H264 NALs which then I send to the network destination. But I have an issue with device rotation, namely, regardless of device orientation I always get 750x1334 frames (like device was held in portrait mode). And so is the content of the frames - if the device is being held in portrait, image displayed is correct, hovewer in landscape frames are rotated 90/270 degrees.


I spent some time Googling, and now I know that I can use CMGetAttachment on video frames send from ReplayKit and get RPVideoSampleOrientationKey, which has correct orientation set. This works fine, and now I'd like to rotate landscape frames properly, for example into 1334x750. And here lies the problem, for I searched Video Toolbox (especially Pixel Transfer stuff) and only found how to set clear aperture and scaling, but no rotation reference.


So right now, I am developing the solution of using vImage to take image from one CVImageBuffer and write it, rotated, to another. But this is hardly ideal solution - maybe there is VT-way of doing simple 90-degrees rotation?


Regards

Michal

Replies  6
Boosts  0
Views  3.2k
Participants  4
danielniemeyer OP
Nov ’17

+ 1 Here, did you find any solutions?

0
MikeAlpha OP
Nov ’17

Alas, not just by using VT - I am locking CVPixelBuffers, then taking plane addresses (it is 420f, with interleaved chromas) and then using Accelerate/vImage multiples-of-90 degrees rotation functions to rotate planes. Rotated frame is then fed to compression and it works OK for me. Michal

0
Charvel OP
May ’18

I am looking for answer for this question, and your answer is very good, but, when I use vImageRotate90_ARGB8888 this system fuction to rotate videobuffer which replayKit send, it crash on vImageRotate90_ARGB8888 function.


So, can you replay me what method you handle.


can give us a demo function ?

0
michelle211 OP
Nov ’19

< Has anyone got around this crash or figured out a quick way to locate the buffer, so far I haven't found a way to change

the orientation in processSampleBuffer


and this code crashes


vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))


let flags = CVPixelBufferLockFlags(rawValue: 0)

guard kCVReturnSuccess == CVPixelBufferLockBaseAddress(srcPixelBuffer, flags) else {

return nil

}

defer { CVPixelBufferUnlockBaseAddress(srcPixelBuffer, flags) }


guard let srcData = CVPixelBufferGetBaseAddress(srcPixelBuffer) else {

print("Error: could not get pixel buffer base address")

return nil

}

let sourceWidth = CVPixelBufferGetWidth(srcPixelBuffer)

let sourceHeight = CVPixelBufferGetHeight(srcPixelBuffer)

var destWidth = sourceHeight

var destHeight = sourceWidth

var color = UInt8(0)


if factor % 2 == 0 {

destWidth = sourceWidth

destHeight = sourceHeight

}


let srcBytesPerRow = CVPixelBufferGetBytesPerRow(srcPixelBuffer)

var srcBuffer = vImage_Buffer(data: srcData,

height: vImagePixelCount(sourceHeight),

width: vImagePixelCount(sourceWidth),

rowBytes: srcBytesPerRow)


let destBytesPerRow = destWidth*4

guard let destData = malloc(destHeight*destBytesPerRow) else {

print("Error: out of memory")

return nil

}

var destBuffer = vImage_Buffer(data: destData,

height: vImagePixelCount(destHeight),

width: vImagePixelCount(destWidth),

rowBytes: destBytesPerRow)


let error = vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))

if error != kvImageNoError {

print("Error:", error)

free(destData) >

0
michelle211 OP
Nov ’19

Hi


Ajy chance you can share your code for rotating the buffer for landscapee.

0
michelle211 OP
Nov ’19

Any chance you can share your code for rotating the buffer for landscape

0
Frame orientation when compressing to H264 using Video Toolbox?
First post date Last post date
 
 
Q