Is it possible to set up a AVCaptureSession in a way that it will deliver 32-bit depth data (instead of 16-bit) during a photo capture? I configured the AVCapturePhotoOutput and the AVCapturePhotoSettings to deliver depth data. And it works: my delegate receives a AVDepthData block… containing 16-bit depth data. I tried setting the AVCaptureDevice's activeDepthDataFormat to a 32-bit format, but the format of the delivered AVDepthData is still only 16-bit—regardless of which format I set on the device. For video capture using an AVCaptureDepthDataOutput this seem to work, just not for an AVCapturePhotoOutput. Any hints are appreciated. 🙂

I set up my AVCaptureSession for photo capture with depth data. In my AVCapturePhotoCaptureDelegate I get the AVCapturePhoto of the capture that contains the depth data. I call fileDataRepresentation() on it and later use a PHAssetCreationRequest to save the image (including the depth data) to a new asset in Photos. When loading the image and its depth data later again, the depth data seemed compressed. I observe some heavy quantization of the data. Is there a way to avoid this compression? Do I need to use specific settings or even a different API for exporting the image?

When setting up a CIContext, one can specify the workingColorSpace. The color space also specifies which gamma curve is used (usually sRGB or linear). When not explicitly setting a color space, Core Image uses a linear curve. It also says this in the (pretty outdated) Core Image Programming Guide - https://developer.apple.com/library/archive/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_advanced_concepts/ci.advanced_concepts.html#//apple_ref/doc/uid/TP30001185-CH9-SW14: By default, Core Image assumes that processing nodes are 128 bits-per-pixel, linear light, premultiplied RGBA floating-point values that use the GenericRGB color space. Now I'm wondering if this makes sense in most scenarios. For instance, if I blur a checkerboard patter with a CIGaussianBlur filter with a default CIContext, I get a different result than when using a non-linear sRGB color space. See here - https://www.icloud.com/keynote/0FLvnwEPx-dkn95dMorENGa0w#Presentation. White gets clearly more weight than black with linear gamma. Which makes sense, I suppose. But I find that the non-linear (sRGB) result looks "more correct". What are best practices here? When should the gamma curve be a consideration?

What is the correct way to set up Core Image for processing (and preserving) wide gamut images? I understand that there are four options for the workingColorSpace: displayP3, extendedLinearDisplayP3, extendedSRGB, and extendedLinearSRGB. While I understand the implications of all of them (linear vs. sRGB gamma curve and Display P3 vs. sRGB primaries), I don't know which is recommended or best practice to use. Also, might there be compatibility issues with the built-in CI filters? Do they make assumptions about the working color space? Further, what's the recommended color space to use for the rendering destination? I assume Display P3 since it's also the color space of photos taken with the iPhone camera... Considering the workingFormat: While I understand that it makes sense to use a 16-bit float type format (RGBAh) for extended range, it also seems very costly. Would it be somehow possible (and advisable) set up the CIContext to use an 8-bit format while still preserving wide gamut? Are there differences or special considerations for the different platforms for both, color space and format? (Sorry for the many questions, but they seem all related…)

One of the announcements from WWDC 2020 was that Family Sharing will be available for IAP and subscriptions: And in addition to shared family app purchases, the App Store now supports Family Sharing for subscriptions and in-app purchases. This is great for developers who offer content for the whole family to enjoy. How can we enable this and is it only available in iOS/iPadOS 14?

In his talk, David mentioned the updated documentation for built-in Core Image filters, which got me very excited. However, I was not able to find it online or in Xcode. I just found out that it's only visible when switching the language to Objective-C (for instance here - https://developer.apple.com/documentation/coreimage/cifilter/3228331-gaussianblurfilter?language=objc). From the screenshot in the presentation it looks like it should be available for Swift as well, though. It also seems that very few filters are documented yet. Is support for Swift and more documentation coming before release? That would be very helpful!

In his talk "Build Metal-based Core Image kernels with Xcode", David presents the build phases necessary to compile Core Image Metal files into Metal libraries, that can be used to instantiate CIKernels. There is a 1-to-1 mapping between .ci.metal file and .ci.metallib file. I also found that the Metal linker doesn't allow to link more then one .air file into one library when building for Core Image. This works fine until I want to have some common code (such as math functions) extracted into another file to be used by multiple kernels. As soon as I have two color kernels (that get concatenated during filter execution) that use the same shared functions, the runtime Metal compiler crashes (I assume because of duplicate symbols in the merged libraries). Is there a good way to extract common functionality to be usable by multiple kernels in a pipeline?

From the iOS 13 release notes: Metal CIKernel instances support arguments with arbitrarily structured data. How does this work? Is there any example code for this? So far I was only able to pass float-typed literals, CIVectors, NSNumbers , CIImages, and CISamplers into kernels as arguments when calling apply.

Some of the filters that can be created using the CIFilterBuiltins extensions cause runtime exceptions when assigning an inputImage to them: NSInvalidArgumentException: "-[CIComicEffect setInputImage:]: unrecognized selector sent to instance ..." I tried a few and found this to be the case for CIFilter.comicEffect(), CIFilter.cmykHalftone(), and CIFilter.pointillize() (probably more). Other filters like CIFilter.gaussianBlur() work fine. This happens in Xcode 11.5 and Xcode 12 beta 2 on iOS and macOS. I already filed feedback for this (FB8013603).

Extending the PencilKit APIs to expose the inner data structures was a great move! And for apps like the "Handwriting Tutor" sample app this is enough. However, I'd love to implement a custom drawing engine based on PKDrawing since it provides a lot of functionality out of the box (user interaction through PKCanvasView, de-/serialization, spline interpolation). But for a custom renderer, two key parts are missing: Custom inks (FB8261616), so we can define custom brushes that render differently than the three system styles. Detecting changes while the user is drawing (FB8261554), otherwise we can't draw their current stroke on screen. I know this was mentioned here before, but I wanted to emphasize that those two features would enable us to implement a full custom render engine based on PencilKit. Thanks for considering!

I have the following setup: An AVCaptureSession with a AVCaptureVideoDataOutput delivers video frames from the camera. OpenGL textures are created from the CVPixelBuffers using a CVOpenGLESTextureCache. Some OpenGL-based image processing is performed on the frames (with many intermediate steps) in a separate queue. The final texture of the processing pipeline is rendered into a CAEAGLLayer on the main thread (with proper context and share group handling). This worked very well up to iOS 13. Now in iOS 14 the AVCaptureVideoDataOutput suddenly stops delivering new frames (to the delegate) after ~4 sec. of capture—without any warning or log message. Some observations: The AVCaptureSession is still running (isRunning is true, isInterrupted is false). All connections between the camera device and output are still there and active. The capture indicator (green circle in the status bar, new in iOS 14) is still there. The output's delegate does not report any frame drops. When I perform an action that causes the session to be re-configured (like switching to the front camera), the output will start delivering frames again for ~4 sec. and then stop again. When I don't process and display the frames, the output continues to deliver frames without interruption. I'm debugging this for a while now and I'm pretty clueless. Any hints or ideas on what might cause this behavior now in iOS 14 are much appreciated! 🙂

I want to play a video and process it using Core Image filters. I have the following setup: the video as an AVAsset an AVPlayer for controlling the video playback the AVAsset is passed via AVPlayerItem to the player the Core Image filters are applied using an AVMutableVideoComposition that was initialized with init(asset:applyingCIFiltersWithHandler:) and assigned to the player item an AVPlayerItemVideoOutput is used to get the processed frames using copyPixelBuffer(forItemTime:itemTimeForDisplay:) This works as expected. However, I'm observing strange behavior when seeking backwards on the player using seek(to:completionHandler:): the AVPlayerItemVideoOutput suddenly stops delivering new pixel buffers and hasNewPixelBuffer(forItemTime:) returns false. But this only happens when the filters applied in the video composition are a bit more expensive. When using a very simple filter or no filter at all, seeking works as expected. Inspired by this technical note - https://developer.apple.com/library/archive/qa/qa1966/_index.html I found a workaround be re-assigning the composition to the player item after the seek finished, but this feels very hacky—especially since it's working for simple filter pipelines. Did anybody encounter this before? Is there anything I need to configure differently?

In the "Explore Core Image kernel improvements" session, David mentioned that it is now possible to compile [[stitchable]] CI kernels at runtime. However, I fail to get it working. The kernel requires the #import of <CoreImage/CoreImage.h> and linking against the CoreImage Metal library. But I don't know how to link against the library when compiling my kernel at runtime. Also, according to the Metal Best Practices Guide, "the #include directive is not supported at runtime for user files." Any guidance on how the runtime compilation works is much appreciated! 🙂

The new VNGeneratePersonSegmentationRequest is a stateful request, i.e. it keeps state and improves the segmentation mask generation for subsequent frames. There is also the new CIPersonSegmentationFilter as a convenient way for using the API with Core Image. But since the Vision request is stateful, I was wondering how this is handled by the Core Image filter. Does the filter also keep state between subsequent calls? How is the "The request requires the use of CMSampleBuffers with timestamps as input" requirement of VNStatefulRequest ensured?

Our account holder got the message that we can now use Xcode Cloud via the beta. However, the developers in our organization are not able to access any Xcode Cloud features. Creating a workflow fails with the message This operation couldn’t be completed. Is there any additional setup required to get Xcode Cloud running on a developer machine?