6 Replies
      Latest reply on Nov 26, 2019 3:32 PM by michelle211
      MikeAlpha Level 3 Level 3 (270 points)

        Hello

         

        I am using ReplayKit to record iOS screen (system-wide recording, started from Control Center) on iPhone 7/iOS 11.1. So far so good, I am getting frames, pushing them into Video Toolbox, then getting H264 NALs which then I send to the network destination. But I have an issue with device rotation, namely, regardless of device orientation I always get 750x1334 frames (like device was held in portrait mode). And so is the content of the frames - if the device is being held in portrait, image displayed is correct, hovewer in landscape frames are rotated 90/270 degrees.

         

        I spent some time Googling, and now I know that I can use CMGetAttachment on video frames send from ReplayKit and get RPVideoSampleOrientationKey, which has correct orientation set. This works fine, and now I'd like to rotate landscape frames properly, for example into 1334x750. And here lies the problem, for I searched Video Toolbox (especially Pixel Transfer stuff) and only found how to set clear aperture and scaling, but no rotation reference.

         

        So right now, I am developing the solution of using vImage to take image from one CVImageBuffer and write it, rotated, to another. But this is hardly ideal solution - maybe there is VT-way of doing simple 90-degrees rotation?

         

        Regards

        Michal

        • Re: Frame orientation when compressing to H264 using Video Toolbox?
          danielniemeyer Level 1 Level 1 (0 points)

          + 1 Here, did you find any solutions?

            • Re: Frame orientation when compressing to H264 using Video Toolbox?
              MikeAlpha Level 3 Level 3 (270 points)

              Alas, not just by using VT - I am locking CVPixelBuffers, then taking plane addresses (it is 420f, with interleaved chromas) and then using Accelerate/vImage multiples-of-90 degrees rotation functions to rotate planes. Rotated frame is then fed to compression and it works OK for me. Michal

                • Re: Frame orientation when compressing to H264 using Video Toolbox?
                  Charvel Level 1 Level 1 (0 points)

                  I am looking for answer for this question, and your answer is very good, but, when I use vImageRotate90_ARGB8888 this system fuction to rotate videobuffer which replayKit send, it crash on vImageRotate90_ARGB8888 function.

                   

                  So, can you replay me what method you handle.

                   

                  can give us a demo function ?

                    • Re: Frame orientation when compressing to H264 using Video Toolbox?
                      michelle211 Level 1 Level 1 (0 points)

                      < Has anyone got around this crash or figured out a quick way to locate the buffer, so far I haven't found a way to change

                      the orientation in processSampleBuffer

                       

                      and this code crashes

                       

                      vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))

                       

                      let flags = CVPixelBufferLockFlags(rawValue: 0)

                        guard kCVReturnSuccess == CVPixelBufferLockBaseAddress(srcPixelBuffer, flags) else {

                          return nil

                        }

                        defer { CVPixelBufferUnlockBaseAddress(srcPixelBuffer, flags) }

                       

                        guard let srcData = CVPixelBufferGetBaseAddress(srcPixelBuffer) else {

                          print("Error: could not get pixel buffer base address")

                          return nil

                        }

                        let sourceWidth = CVPixelBufferGetWidth(srcPixelBuffer)

                        let sourceHeight = CVPixelBufferGetHeight(srcPixelBuffer)

                        var destWidth = sourceHeight

                        var destHeight = sourceWidth

                        var color = UInt8(0)

                       

                        if factor % 2 == 0 {

                          destWidth = sourceWidth

                          destHeight = sourceHeight

                        }

                       

                        let srcBytesPerRow = CVPixelBufferGetBytesPerRow(srcPixelBuffer)

                        var srcBuffer = vImage_Buffer(data: srcData,

                                                      height: vImagePixelCount(sourceHeight),

                                                      width: vImagePixelCount(sourceWidth),

                                                      rowBytes: srcBytesPerRow)

                       

                        let destBytesPerRow = destWidth*4

                        guard let destData = malloc(destHeight*destBytesPerRow) else {

                          print("Error: out of memory")

                          return nil

                        }

                        var destBuffer = vImage_Buffer(data: destData,

                                                       height: vImagePixelCount(destHeight),

                                                       width: vImagePixelCount(destWidth),

                                                       rowBytes: destBytesPerRow)

                       

                        let error = vImageRotate90_ARGB8888(&srcBuffer, &destBuffer, factor, &color, vImage_Flags(0))

                        if error != kvImageNoError {

                          print("Error:", error)

                          free(destData) >

                      • Re: Frame orientation when compressing to H264 using Video Toolbox?
                        michelle211 Level 1 Level 1 (0 points)

                        Hi

                         

                        Ajy chance you can share your code  for rotating the  buffer for landscapee.

                  • Re: Frame orientation when compressing to H264 using Video Toolbox?
                    michelle211 Level 1 Level 1 (0 points)

                    Any chance you can share your code  for rotating the  buffer for landscape