I am trying to iterate over images in Photo Library and extract faces using CIDetector. The images are required to keep their original resolutions. To do so, I taking the following steps:
1- Getting assets given a date interval (usually more than a year)
func loadAssets(from fromDate: Date, to toDate: Date, completion: @escaping ([PHAsset]) -> Void) {
fetchQueue.async {
let authStatus = PHPhotoLibrary.authorizationStatus()
if authStatus == .authorized || authStatus == .limited {
let options = PHFetchOptions()
options.predicate = NSPredicate(format: "creationDate >= %@ && creationDate <= %@", fromDate as CVarArg, toDate as CVarArg)
options.sortDescriptors = [NSSortDescriptor(key: "creationDate", ascending: false)]
let result: PHFetchResult = PHAsset.fetchAssets(with: .image, options: options)
var _assets = [PHAsset]()
result.enumerateObjects { object, count, stop in
_assets.append(object)
}
completion(_assets)
} else {
completion([])
}
}
}
where:
let fetchQueue = DispatchQueue.global(qos: .background)
2- Extracting faces
I then extract face images using:
func detectFaces(in image: UIImage, accuracy: String = CIDetectorAccuracyLow, completion: @escaping ([UIImage]) -> Void) {
faceDetectionQueue.async {
var faceImages = [UIImage]()
let outputImageSize: CGFloat = 200.0 / image.scale
guard let ciImage = CIImage(image: image),
let faceDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: [CIDetectorAccuracy: accuracy]) else { completion(faceImages); return }
let faces = faceDetector.features(in: ciImage) // Crash happens here
let group = DispatchGroup()
for face in faces {
group.enter()
if let face = face as? CIFaceFeature {
let faceBounds = face.bounds
let offset: CGFloat = floor(min(faceBounds.width, faceBounds.height) * 0.2)
let inset = UIEdgeInsets(top: -offset, left: -offset, bottom: -offset, right: -offset)
let rect = faceBounds.inset(by: inset)
let croppedFaceImage = ciImage.cropped(to: rect)
let scaledImage = croppedFaceImage
.transformed(by: CGAffineTransform(scaleX: outputImageSize / croppedFaceImage.extent.width,
y: outputImageSize / croppedFaceImage.extent.height))
faceImages.append(UIImage(ciImage: scaledImage))
group.leave()
} else {
group.leave()
}
}
group.notify(queue: self.faceDetectionQueue) {
completion(faceImages)
}
}
}
where:
private let faceDetectionQueue = DispatchQueue(label: "face detection queue",
qos: DispatchQoS.background,
attributes: [],
autoreleaseFrequency: DispatchQueue.AutoreleaseFrequency.workItem,
target: nil)
I use the following extension to get the image from assets:
extension PHAsset {
var image: UIImage {
autoreleasepool {
let manager = PHImageManager.default()
let options = PHImageRequestOptions()
var thumbnail = UIImage()
let rect = CGRect(x: 0, y: 0, width: pixelWidth, height: pixelHeight)
options.isSynchronous = true
options.deliveryMode = .highQualityFormat
options.resizeMode = .exact
options.normalizedCropRect = rect
options.isNetworkAccessAllowed = true
manager.requestImage(for: self, targetSize: rect.size, contentMode: .aspectFit, options: options, resultHandler: {(result, info) -> Void in
if let result = result {
thumbnail = result
} else {
thumbnail = UIImage()
}
})
return thumbnail
}
}
}
The code works fine for a few (usually less that 50) assets, but for more number of images it crashes at:
let faces = faceDetector.features(in: ciImage) // Crash happens here
I get this error:
validateComputeFunctionArguments:858: failed assertion `Compute Function(ciKernelMain): missing sampler binding at index 0 for [0].'
If I reduce the size of the image fed to detectFaces(:) e.g. 400 px, I can analyze a few hundred images (usually less than 1000) but as I mentioned, using the asset's image in the original size is a requirement. My guess is it has something to do with a memory issue when I try to extract faces with CIDetector. Any idea what this error is about and how I can fix the issue?
Asteroid
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
User for
Post
Replies
Boosts
Views
Activity
I have an AVFoundation-based live camera view. There is a button by which I am calling AVCaptureDevice.showSystemUserInterface(.videoEffects) so that the user can activate the Portrait Effect. I have also opted in by setting "Camera — Opt in for Portrait Effect" to true in info.plist. However, upon tapping on the button I see this screen (The red crossed-off part is the app name):
I am expecting to see something like this:
Do you have any idea why that might be?
I have an AVFoundation-based live camera view. There is a button by which I am calling AVCaptureDevice.showSystemUserInterface(.videoEffects) so that the user can activate the Portrait Effect. I have also opted in by setting "Camera — Opt in for Portrait Effect" to true in info.plist. However, upon tapping on the button I see this screen (The red crossed-off part is the app name):
I am expecting to see something like this:
Do you have any idea why that might be?
I have a camera view based on AVFoundation. Any idea how I can switch to Portrait Effect in iOS's Control Center like SnapChat?
I am using AVFoundation for live camera view. I can get my device from the current video input (of type AVCaptureDeviceInput) like:
let device = videoInput.device
The device's active format has a isPortraitEffectSupported. How can I set the Portrait Effect on and off in live camera view?
I setup the camera like this:
private var videoInput: AVCaptureDeviceInput!
private let session = AVCaptureSession()
private(set) var isSessionRunning = false
private var renderingEnabled = true
private let videoDataOutput = AVCaptureVideoDataOutput()
private let photoOutput = AVCapturePhotoOutput()
private(set) var cameraPosition: AVCaptureDevice.Position = .front
func configureSession() {
sessionQueue.async { [weak self] in
guard let strongSelf = self else { return }
if strongSelf.setupResult != .success {
return
}
let defaultVideoDevice: AVCaptureDevice? = strongSelf.videoDeviceDiscoverySession.devices.first(where: {$0.position == strongSelf.cameraPosition})
guard let videoDevice = defaultVideoDevice else {
print("Could not find any video device")
strongSelf.setupResult = .configurationFailed
return
}
do {
strongSelf.videoInput = try AVCaptureDeviceInput(device: videoDevice)
} catch {
print("Could not create video device input: \(error)")
strongSelf.setupResult = .configurationFailed
return
}
strongSelf.session.beginConfiguration()
strongSelf.session.sessionPreset = AVCaptureSession.Preset.photo
// Add a video input.
guard strongSelf.session.canAddInput(strongSelf.videoInput) else {
print("Could not add video device input to the session")
strongSelf.setupResult = .configurationFailed
strongSelf.session.commitConfiguration()
return
}
strongSelf.session.addInput(strongSelf.videoInput)
// Add a video data output
if strongSelf.session.canAddOutput(strongSelf.videoDataOutput) {
strongSelf.session.addOutput(strongSelf.videoDataOutput)
strongSelf.videoDataOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)]
strongSelf.videoDataOutput.setSampleBufferDelegate(self, queue: strongSelf.dataOutputQueue)
} else {
print("Could not add video data output to the session")
strongSelf.setupResult = .configurationFailed
strongSelf.session.commitConfiguration()
return
}
// Add photo output
if strongSelf.session.canAddOutput(strongSelf.photoOutput) {
strongSelf.session.addOutput(strongSelf.photoOutput)
strongSelf.photoOutput.isHighResolutionCaptureEnabled = true
} else {
print("Could not add photo output to the session")
strongSelf.setupResult = .configurationFailed
strongSelf.session.commitConfiguration()
return
}
strongSelf.session.commitConfiguration()
}
}
func prepareSession(completion: @escaping (SessionSetupResult) -> Void) {
sessionQueue.async { [weak self] in
guard let strongSelf = self else { return }
switch strongSelf.setupResult {
case .success:
strongSelf.addObservers()
if strongSelf.photoOutput.isDepthDataDeliverySupported {
strongSelf.photoOutput.isDepthDataDeliveryEnabled = true
}
if let photoOrientation = AVCaptureVideoOrientation(interfaceOrientation: interfaceOrientation) {
if let unwrappedPhotoOutputConnection = strongSelf.photoOutput.connection(with: .video) {
unwrappedPhotoOutputConnection.videoOrientation = photoOrientation
}
}
strongSelf.dataOutputQueue.async {
strongSelf.renderingEnabled = true
}
strongSelf.session.startRunning()
strongSelf.isSessionRunning = strongSelf.session.isRunning
strongSelf.mainQueue.async {
strongSelf.previewView.videoPreviewLayer.session = strongSelf.session
}
completion(strongSelf.setupResult)
default:
completion(strongSelf.setupResult)
}
}
}
Then to I set isPortraitEffectsMatteDeliveryEnabled like this:
func setPortraitAffectActive(_ state: Bool) {
sessionQueue.async { [weak self] in
guard let strongSelf = self else { return }
if strongSelf.photoOutput.isPortraitEffectsMatteDeliverySupported {
strongSelf.photoOutput.isPortraitEffectsMatteDeliveryEnabled = state
}
}
}
However, I don't see any Portrait Effect in the live camera view! Any ideas why?
I am trying to develop an ARKit iOS app. Is it possible to use Apple Watch to see a live view from ARKit (run from the parent iOS app)?