I have set AVCaptureVideoDataOutput with 10-bit 420 YCbCr sample buffers. I use Core Image to process these pixel buffers for simple scaling/translation. var dstBounds = CGRect.zero dstBounds.size = dstImage.extent.size /* *srcImage is created from sample buffer received from Video Data Output */ _ciContext.render(dstImage, to: dstPixelBuffer!, bounds: dstImage.extent, colorSpace: srcImage.colorSpace ) I then set the color attachments to this dstPixelBuffer using set colorProfile in the app settings (BT.709 or BT.2020). switch colorProfile { case .BT709: CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, .shouldPropagate) CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, .shouldPropagate) CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_709_2, .shouldPropagate) case .HLG2100: CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_2020, .shouldPropagate) CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_2100_HLG, .shouldPropagate) CVBufferSetAttachment(dstPixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_2020, .shouldPropagate) } These pixel buffers are then vended to AVAssetWriter whose videoSettings is set to recommendedSettings by VDO. But the output seems to be washed out completely, esp. for SDR (BT.709). What am I doing wrong?

I have two CIContexts configured with the following options: let options1:[CIContextOption:Any] = [CIContextOption.cacheIntermediates: false, CIContextOption.outputColorSpace: NSNull(), CIContextOption.workingColorSpace: NSNull()]; let options2:[CIContextOption:Any] = [CIContextOption.cacheIntermediates: false]; And an MTKView with CAMetalLayer configured with HDR output. metalLayer = self.layer as? CAMetalLayer metalLayer?.wantsExtendedDynamicRangeContent = true metalLayer.colorspace = CGColorSpace(name: CGColorSpace.itur_2100_HLG) colorPixelFormat = .bgr10a2Unorm The two context options produce different outputs when input is in BT.2020 pixel buffers. But I believe the outputs shouldn't be different. Because the first option simply disables color management. The second one performs intermediate buffer calculations in sRGB extended linear color space and then converts those buffers to BT.2020 color space in the output.

It looks like [[stitchable]] Metal Core Image kernels fail to get added in the default metal library. Here is my code: class FilterTwo: CIFilter { var inputImage: CIImage? var inputParam: Float = 0.0 static var kernel: CIKernel = { () -> CIKernel in let url = Bundle.main.url(forResource: "default", withExtension: "metallib")! let data = try! Data(contentsOf: url) let kernelNames = CIKernel.kernelNames(fromMetalLibraryData: data) NSLog("Kernels \(kernelNames)") return try! CIKernel(functionName: "secondFilter", fromMetalLibraryData: data) //<-- This fails! }() override var outputImage : CIImage? { guard let inputImage = inputImage else { return nil } return FilterTwo.kernel.apply(extent: inputImage.extent, roiCallback: { (index, rect) in return rect }, arguments: [inputImage]) } } Here is the Metal code: using namespace metal; [[ stitchable ]] half4 secondFilter (coreimage::sampler inputImage, coreimage::destination dest) { float2 srcCoord = inputImage.coord(); half4 color = half4(inputImage.sample(srcCoord)); return color; } And here is the usage: class ViewController: UIViewController { override func viewDidLoad() { super.viewDidLoad() // Do any additional setup after loading the view. let filter = FilterTwo() filter.inputImage = CIImage(color: CIColor.red) let outputImage = filter.outputImage! NSLog("Output \(outputImage)") } } And the output: StitchableKernelsTesting/FilterTwo.swift:15: Fatal error: 'try!' expression unexpectedly raised an error: Error Domain=CIKernel Code=1 "(null)" UserInfo={CINonLocalizedDescriptionKey=Function does not exist in library data. …•∆} Kernels [] reflect Function 'secondFilter' does not exist.

I have imported two metal files and defined two stitchable Metal Core Image kernels, one of them being CIColorKernel and other being CIKernel. As outlined in the WWDC video, I need to add a flag -framework CoreImage to other Metal Linker flags. Unfortunately, Xcode 15 puts a double quotes around this and generates an error metal: error: unknown argument: '-framework CoreImage'. So I built without this flag and it works for the first kernel that was added. The other kernel is never added to metal.defaultlib and fails to load. How do I get it working? class SobelEdgeFilterHDR: CIFilter { var inputImage: CIImage? var inputParam: Float = 0.0 static var kernel: CIKernel = { () -> CIKernel in let url = Bundle.main.url(forResource: "default", withExtension: "metallib")! let data = try! Data(contentsOf: url) let kernelNames = CIKernel.kernelNames(fromMetalLibraryData: data) NSLog("Kernels \(kernelNames)") return try! CIKernel(functionName: "sobelEdgeFilterHDR", fromMetalLibraryData: data) }() override var outputImage : CIImage? { guard let inputImage = inputImage else { return nil } return SobelEdgeFilterHDR.kernel.apply(extent: inputImage.extent, roiCallback: { (index, rect) in return rect }, arguments: [inputImage]) } }

I have been allocating pixel buffers from CVPixelBufferPool and the code has been adapted from older various Apple sample codes such as RosyWriter. I see direct API such as CVPixelBufferCreate are highly performant and rarely cause frame drops as opposed to allocating from pixel buffer pool where I regularly get frame drops. Is this a known issue or a bad use of API? Here is the code for creating pixel buffer pool: private func createPixelBufferPool(_ width: Int32, _ height: Int32, _ pixelFormat: FourCharCode, _ maxBufferCount: Int32) -> CVPixelBufferPool? { var outputPool: CVPixelBufferPool? = nil let sourcePixelBufferOptions: NSDictionary = [kCVPixelBufferPixelFormatTypeKey: pixelFormat, kCVPixelBufferWidthKey: width, kCVPixelBufferHeightKey: height, kCVPixelFormatOpenGLESCompatibility: true, kCVPixelBufferIOSurfacePropertiesKey: [:] as CFDictionary] let pixelBufferPoolOptions: NSDictionary = [kCVPixelBufferPoolMinimumBufferCountKey: maxBufferCount] CVPixelBufferPoolCreate(kCFAllocatorDefault, pixelBufferPoolOptions, sourcePixelBufferOptions, &outputPool) return outputPool } private func createPixelBufferPoolAuxAttributes(_ maxBufferCount: size_t) -> NSDictionary { // CVPixelBufferPoolCreatePixelBufferWithAuxAttributes() will return kCVReturnWouldExceedAllocationThreshold if we have already vended the max number of buffers return [kCVPixelBufferPoolAllocationThresholdKey: maxBufferCount] } private func preallocatePixelBuffersInPool(_ pool: CVPixelBufferPool, _ auxAttributes: NSDictionary) { // Preallocate buffers in the pool, since this is for real-time display/capture var pixelBuffers: [CVPixelBuffer] = [] while true { var pixelBuffer: CVPixelBuffer? = nil let err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes(kCFAllocatorDefault, pool, auxAttributes, &pixelBuffer) if err == kCVReturnWouldExceedAllocationThreshold { break } assert(err == noErr) pixelBuffers.append(pixelBuffer!) } pixelBuffers.removeAll() } And here is the usage: bufferPool = createPixelBufferPool(outputDimensions.width, outputDimensions.height, outputPixelFormat, Int32(maxRetainedBufferCount)) if bufferPool == nil { NSLog("Problem initializing a buffer pool.") success = false break bail } bufferPoolAuxAttributes = createPixelBufferPoolAuxAttributes(maxRetainedBufferCount) preallocatePixelBuffersInPool(bufferPool!, bufferPoolAuxAttributes!) And then creating pixel buffers from pool err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes( kCFAllocatorDefault, bufferPool!, bufferPoolAuxAttributes, &dstPixelBuffer ) if err == kCVReturnWouldExceedAllocationThreshold { // Flush the texture cache to potentially release the retained buffers and try again to create a pixel buffer err = CVPixelBufferPoolCreatePixelBufferWithAuxAttributes( kCFAllocatorDefault, bufferPool!, bufferPoolAuxAttributes, &dstPixelBuffer ) } if err != 0 { if err == kCVReturnWouldExceedAllocationThreshold { NSLog("Pool is out of buffers, dropping frame") } else { NSLog("Error at CVPixelBufferPoolCreatePixelBuffer %d", err) } break bail } When used with AVAssetWriter, I see lot of frame drops caused due to kCVReturnWouldExceedAllocationThreshold error. No frame drops are seen when I directly allocate the pixel buffer without using a pool: CVPixelBufferCreate(kCFAllocatorDefault, Int(dimensions.width), Int(dimensions.height), outputPixelFormat, sourcePixelBufferOptions, &dstPixelBuffer) What could be the cause?

I am currently using CoreImage to process YCbCr422/420 10-bit pixel buffers but it is lacking performance at high frame rates so I decided to switch to Metal. But with Metal I am getting even worse performance. I am loading both the Luma (Y) and Chroma (CbCr) textures in 16-bit format as follows: let pixelFormatY = MTLPixelFormat.r16Unorm let pixelFormatUV = MTLPixelFormat.rg16Unorm renderPassDescriptorY!.colorAttachments[0].texture = texture; renderPassDescriptorY!.colorAttachments[0].loadAction = .clear; renderPassDescriptorY!.colorAttachments[0].clearColor = MTLClearColor(red: 0.0, green: 0.0, blue: 0.0, alpha: 1.0) renderPassDescriptorY!.colorAttachments[0].storeAction = .store; renderPassDescriptorCbCr!.colorAttachments[0].texture = texture; renderPassDescriptorCbCr!.colorAttachments[0].loadAction = .clear; renderPassDescriptorCbCr!.colorAttachments[0].clearColor = MTLClearColor(red: 0.0, green: 0.0, blue: 0.0, alpha: 1.0) renderPassDescriptorCbCr!.colorAttachments[0].storeAction = .store; // Vertices and texture coordinates for Metal shader let vertices:[AAPLVertex] = [AAPLVertex(position: vector_float2(-1.0, -1.0), texCoord: vector_float2( 0.0 , 1.0)), AAPLVertex(position: vector_float2(1.0, -1.0), texCoord: vector_float2( 1.0, 1.0)), AAPLVertex(position: vector_float2(-1.0, 1.0), texCoord: vector_float2( 0.0, 0.0)), AAPLVertex(position: vector_float2(1.0, 1.0), texCoord: vector_float2( 1.0, 0.0)) ] let commandBuffer = commandQueue!.makeCommandBuffer() if let commandBuffer = commandBuffer { let renderEncoderY = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptorY!) renderEncoderY?.setRenderPipelineState(pipelineStateY!) renderEncoderY?.setVertexBytes(vertices, length: vertices.count * MemoryLayout<AAPLVertex>.stride, index: 0) renderEncoderY?.setFragmentTexture(CVMetalTextureGetTexture(lumaTexture!), index: 0) renderEncoderY?.setViewport(MTLViewport(originX: 0, originY: 0, width: Double(dstWidthY), height: Double(dstHeightY), znear: 0, zfar: 1)) renderEncoderY?.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, instanceCount: 1) renderEncoderY?.endEncoding() let renderEncoderCbCr = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptorCbCr!) renderEncoderCbCr?.setRenderPipelineState(pipelineStateCbCr!) renderEncoderCbCr?.setVertexBytes(vertices, length: vertices.count * MemoryLayout<AAPLVertex>.stride, index: 0) renderEncoderCbCr?.setFragmentTexture(CVMetalTextureGetTexture(chromaTexture!), index: 0) renderEncoderCbCr?.setViewport(MTLViewport(originX: 0, originY: 0, width: Double(dstWidthUV), height: Double(dstHeightUV), znear: 0, zfar: 1)) renderEncoderCbCr?.drawPrimitives(type: .triangleStrip, vertexStart: 0, vertexCount: 4, instanceCount: 1) renderEncoderCbCr?.endEncoding() commandBuffer.commit() } And here is shader code: vertex MappedVertex vertexShaderYCbCrPassthru ( constant Vertex *vertices [[ buffer(0) ]], unsigned int vertexId [[vertex_id]] ) { MappedVertex out; Vertex v = vertices[vertexId]; out.renderedCoordinate = float4(v.position, 0.0, 1.0); out.textureCoordinate = v.texCoord; return out; } fragment half fragmentShaderYPassthru ( MappedVertex in [[ stage_in ]], texture2d<float, access::sample> textureY [[ texture(0) ]] ) { constexpr sampler s(s_address::clamp_to_edge, t_address::clamp_to_edge, min_filter::linear, mag_filter::linear); float Y = float(textureY.sample(s, in.textureCoordinate).r); return half(Y); } fragment half2 fragmentShaderCbCrPassthru ( MappedVertex in [[ stage_in ]], texture2d<float, access::sample> textureCbCr [[ texture(0) ]] ) { constexpr sampler s(s_address::clamp_to_edge, t_address::clamp_to_edge, min_filter::linear, mag_filter::linear); float2 CbCr = float2(textureCbCr.sample(s, in.textureCoordinate).rg); return half2(CbCr); } Is there anything fundamentally wrong in the code that makes it slow?

I have the following code adapted from AVCamManual sample code to set white balance. I still see crash reports in analytics where exception is raised in setting WB: *** -[AVCaptureDevice temperatureAndTintValuesForDeviceWhiteBalanceGains:] whiteBalanceGains contain an out-of-range value - red, green, and blue gain Here is my code, it is not clear how things are turning out of range. public func normalizedGains(gains:AVCaptureDevice.WhiteBalanceGains) -> AVCaptureDevice.WhiteBalanceGains { var g = gains if let device = videoDevice { g.redGain = max(1.0, g.redGain) g.blueGain = max(1.0, g.blueGain) g.greenGain = max(1.0, g.greenGain) g.redGain = min(device.maxWhiteBalanceGain, g.redGain) g.blueGain = min(device.maxWhiteBalanceGain, g.blueGain) g.greenGain = min(device.maxWhiteBalanceGain, g.greenGain) } return g } And my code to set WB: public func setTemperatureAndTint( colorTemperature:Float?, tint:Float?) { if let device = videoDevice { var tint = tint var colorTemperature = colorTemperature if colorTemperature == nil { colorTemperature = device.temperatureAndTintValues(for: device.deviceWhiteBalanceGains).temperature } if tint == nil { tint = device.temperatureAndTintValues(for: device.deviceWhiteBalanceGains).tint } let temperatureTint = AVCaptureDevice.WhiteBalanceTemperatureAndTintValues(temperature: colorTemperature!, tint: tint!) NSLog("Setting tint \(temperatureTint.tint)") do { try device.lockForConfiguration() device.setWhiteBalanceModeLocked(with: normalizedGains(gains: device.deviceWhiteBalanceGains(for: temperatureTint)) , completionHandler: nil) device.unlockForConfiguration() wbLockedtoGray = false } catch { NSLog("Unable to change White balance gain \(error)") } } } Is there anything I am doing wrong?

I have CVPixelBuffers in YCbCr (422 or 420 biplanar 10 bit video range) coming from camera. I see vImage framework is sophisticated enough to handle a variety of image formats (including pixel buffers in various YCbCr formats). I was looking to compute histograms for both Y (luma) and RGB. For the 8 bit YCbCr samples, I could use this code to compute histogram of Y component. CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0) let baseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0) let height = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0) let width = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0) CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly) var buffer = vImage_Buffer(data: baseAddress, height: vImagePixelCount( height), width: vImagePixelCount(width), rowBytes: bytesPerRow) alphaBin.withUnsafeMutableBufferPointer { alphaPtr in let error = vImageHistogramCalculation_Planar8(&buffer, alphaPtr.baseAddress!, UInt32(kvImageNoFlags)) guard error == kvImageNoError else { fatalError("Error calculating histogram luma: \(error)") } } How does one implement the same for 10 bit HDR pixel buffers, preferably using new iOS 16 vImage APIs that provide lot more flexibility (for instance, getting RGB histogram as well from YCbCr sample without explicitly performing pixel format conversion)?

It seems AVAssetWriter is rejecting CVPixelBuffers with error -12743 when appending NSData for kCVImageBufferAmbientViewingEnvironmentKey for HDR videos. Here is my code: var ambientViewingEnvironment:CMFormatDescription.Extensions.Value? var ambientViewingEnvironmentData:NSData? ambientViewingEnvironment = sampleBuffer.formatDescription?.extensions[.ambientViewingEnvironment] let plist = ambientViewingEnvironment?.propertyListRepresentation ambientViewingEnvironmentData = plist as? NSData And then attaching this data, CVBufferSetAttachment(dstPixelBuffer, kCVImageBufferAmbientViewingEnvironmentKey, ambientViewingEnvironmentData! as CFData, .shouldPropagate) No matter what I do, including copying the attachment from sourcePixelBuffer to destinationPixelBuffer as it is, the error remains! var attachmentMode:CVAttachmentMode = .shouldPropagate let attachment = CVBufferCopyAttachment(sourcePixelBuffer!, kCVImageBufferAmbientViewingEnvironmentKey, &attachmentMode) NSLog("Attachment \(attachment!), mode \(attachmentMode)") CVBufferSetAttachment(dstPixelBuffer, kCVImageBufferAmbientViewingEnvironmentKey, attachment!, attachmentMode) I need to know if there is anything wrong in the way metadata is copied.

I need to know correct color conversion matrices for converting YCbCr422 and YCbCr420 10 bit video range sample buffers (BT.2020 color space) to RGB. AVFoundation framework mentions AVVideoYCbCrMatrix_ITU_R_2020 which is a string constant. But I need to know the full matrix that can be used to perform color conversion. I have this matrix for full range BT2020, not sure if this is correct and what is the correct way to adapt it to video range. let colorMatrixBT2020_fullRange = ColorConversion(matrix: matrix_float3x3(columns: (simd_float3(1.0, 1.0, 1.0), simd_float3(0.000, -0.11156702/0.6780, 1.8814), simd_float3(1.4746, -0.38737742/0.6780, 0.000))), offset: vector_float3(0.0, -0.5, -0.5))

The native camera app on iPhone 15 can record videos directly in external hard drive. Is there an API to achieve the same in Photos framework?

I want to know the correct way to enable Apple log on AVCaptureDevice. I understand one can query device formats, query whether a format supports AVCaptureColorSpace_AppleLog, and set the format and set the activeColorSpace to this value. Is that the way to enable Log?

I noticed on iOS 17 that calling AppTransaction.shared fails with following errors. Here is the code: let result: VerificationResult<AppTransaction> = try await AppTransaction.shared NSLog("Result \(result)") It does not hit the NSLog statement and instead throws the following error logs. Error getting app transaction: Error Domain=ASDErrorDomain Code=500 "(null)" UserInfo={NSUnderlyingError=0x281815050 {Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo={NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit-sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code}}} Received error that does not have a corresponding StoreKit Error: Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo= {NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit- sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code} Received error that does not have a corresponding StoreKit Error: Error Domain=ASDErrorDomain Code=500 "(null)" UserInfo=.{NSUnderlyingError=0x281815050 {Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo={NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit-sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code}}}

I am trying to migrate my app from paid to freemium and am facing several issues and doubts. Specifically, I am trying to use StoreKit2 AppTransaction API but I am not averse to using StoreKit if my problems are not solved by StoreKit2: Here are my questions: AppTransaction/Receipt on launch: I see on launch the AppTransaction.shared call fails on the sandbox initially. That means it's possible that on user's who have purchased the app previously, the AppTransaction (or appStoreReceipt in original StoreKit) may not be available when the user downloads or updates the app? That means I will need to ask every user to authenticate with AppStore to refresh the receipt/AppTransaction? Volume Purchase Users: I see StoreKit2 is not advised for volume purchases on the Apple website. I am not sure why that is the case, but does that mean AppTransaction will not be available for users who made Volume purchases under VPP? Is the flow to validate VPP users different? If StoreKit 2 can not be used, can the original StoreKit API help here, or nothing can be of help here?

I am looking to move from paid app to fremium without upsetting my existing users. I see WWDC2022 session where new fields introduced in iOS 16 are used to extract original application version user used to purchase the app. While my app supports iOS 14 and above, I am willing to sacrifice iOS 14 and go iOS 15 and above as StoreKit2 requires iOS 15 at the minimum. The code below is however only valid for iOS 16. I need to know what is the best way out for iOS 15 devices if I am using StoreKit2? If it is not possible in StoreKit2, then how much is the work involved in original StoreKit API(because in that case I can handle for iOS 14 as well)?