2 Replies
      Latest reply on Jan 6, 2020 9:02 PM by shiwani18
      rmeinl Level 1 Level 1 (0 points)

        I am trying to rectify the lens distortion on the Depth Data from the TrueDepth camera. I reproduced the code from AVCameraCalibrationData.h and then built a function that loops over the CVPixelBuffer while replacing each point. For the first point (0.0) the function gives me a negative value which causes the app to crash.

         

            func lensDistortionPoint(for point: CGPoint, lookupTable: Data, distortionOpticalCenter opticalCenter: CGPoint, imageSize: CGSize) -> CGPoint {
                // The lookup table holds the relative radial magnification for n linearly spaced radii.
                // The first position corresponds to radius = 0
                // The last position corresponds to the largest radius found in the image.
                
                // Determine the maximum radius.
                let delta_ocx_max = Float(max(opticalCenter.x, imageSize.width  - opticalCenter.x))
                let delta_ocy_max = Float(max(opticalCenter.y, imageSize.height - opticalCenter.y))
                let r_max = sqrt(delta_ocx_max * delta_ocx_max + delta_ocy_max * delta_ocy_max)
                
                // Determine the vector from the optical center to the given point.
                let v_point_x = Float(point.x - opticalCenter.x)
                let v_point_y = Float(point.y - opticalCenter.y)
                
                // Determine the radius of the given point.
                let r_point = sqrt(v_point_x * v_point_x + v_point_y * v_point_y)
                
                // Look up the relative radial magnification to apply in the provided lookup table
                let magnification: Float = lookupTable.withUnsafeBytes { (lookupTableValues: UnsafePointer<Float>) in
                    let lookupTableCount = lookupTable.count / MemoryLayout<Float>.size
                    
                    if r_point < r_max {
                        // Linear interpolation
                        let val   = r_point * Float(lookupTableCount - 1) / r_max
                        let idx   = Int(val)
                        let frac  = val - Float(idx)
                        
                        let mag_1 = lookupTableValues[idx]
                        let mag_2 = lookupTableValues[idx + 1]
                        
                        return (1.0 - frac) * mag_1 + frac * mag_2
                    } else {
                        return lookupTableValues[lookupTableCount - 1]
                    }
                }
                
                // Apply radial magnification
                let new_v_point_x = v_point_x + magnification * v_point_x
                let new_v_point_y = v_point_y + magnification * v_point_y
                
                // Construct output
                return CGPoint(x: opticalCenter.x + CGFloat(new_v_point_x), y: opticalCenter.y + CGFloat(new_v_point_y))
            }
            
            private func rectifyDepthData(avDepthData: AVDepthData, image: UIImage) -> CVPixelBuffer? {
                guard
                    let distortionLookupTable = avDepthData.cameraCalibrationData?.lensDistortionLookupTable,
                    let distortionCenter = avDepthData.cameraCalibrationData?.lensDistortionCenter else {
                        return nil
                }
                
                let originalDepthDataMap = avDepthData.depthDataMap
                let width = CVPixelBufferGetWidth(originalDepthDataMap)
                let height = CVPixelBufferGetHeight(originalDepthDataMap)
                let scaledCenter = CGPoint(x: (distortionCenter.x / CGFloat(image.size.height)) * CGFloat(width), y: (distortionCenter.y / CGFloat(image.size.width)) * CGFloat(height))
                CVPixelBufferLockBaseAddress(originalDepthDataMap, CVPixelBufferLockFlags(rawValue: 0))
                
                var maybePixelBuffer: CVPixelBuffer?
                let status = CVPixelBufferCreate(nil, width, height, avDepthData.depthDataType, nil, &maybePixelBuffer)
                
                assert(status == kCVReturnSuccess && maybePixelBuffer != nil);
                
                guard let rectifiedPixelBuffer = maybePixelBuffer else {
                    return nil
                }
                
                CVPixelBufferLockBaseAddress(rectifiedPixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
                guard let address = CVPixelBufferGetBaseAddress(rectifiedPixelBuffer) else {
                    return nil
                }
                for y in 0 ..< height{
                    let rowData = CVPixelBufferGetBaseAddress(originalDepthDataMap)! + y * CVPixelBufferGetBytesPerRow(originalDepthDataMap)
                    let data = UnsafeBufferPointer(start: rowData.assumingMemoryBound(to: Float32.self), count: width)
                    
                    for x in 0 ..< width{
                        let oldPoint = CGPoint(x: x, y: y)
                        let newPoint = lensDistortionPoint(for: oldPoint, lookupTable: distortionLookupTable, distortionOpticalCenter: scaledCenter, imageSize: CGSize(width: width, height: height) )
                        let val = data[x]
                        
                        let newRow = address + Int(newPoint.y) * CVPixelBufferGetBytesPerRow(rectifiedPixelBuffer)
                        let newData = UnsafeMutableBufferPointer(start: newRow.assumingMemoryBound(to: Float32.self), count: width)
                        print(newPoint)
                        newData[Int(newPoint.x)] = val
                    }
                }
                CVPixelBufferUnlockBaseAddress(rectifiedPixelBuffer, CVPixelBufferLockFlags(rawValue: 0))
                CVPixelBufferUnlockBaseAddress(originalDepthDataMap, CVPixelBufferLockFlags(rawValue: 0))
                return rectifiedPixelBuffer
            }
        
        
        • Re: Rectify AVDepthData
          bford Apple Staff Apple Staff (315 points)

          Looks like you're doing a "push" instead of a "pull". You need to flip your thinking.

          You're currently looping over each input location, transforming the coordinate to the appropriate output location and then pushing pixels to the output. This will leave gaps in the output buffer.

           

          Instead, you should loop over all output pixels, do the inverse transformation, clamp the points (so they are within the boundaries of the buffer) and then sample/pull pixels from the input into the output buffer.

           

          Does this make sense?

            • Re: Rectify AVDepthData
              shiwani18 Level 1 Level 1 (0 points)

              Hello bford,
                        i tried to print the value of newData array and found that value of newdata[x] starts with 4 and ends with newData[x]=634 from above code,using  inverse lens distortion lookup table.is this correct ?or should we use the lensDistortionLookupTabel?