I'm trying to detect faces in my iOS camera app, but it doesn't work properly, while it works properly in Camera.app. Notice that:
- The first face isns't detected in my app, only in Camera.app.
- For the third face — the east asian woman — Camera.app correctly draws a rectangle around her face, while my app draws a rectangle that extends far below her face.
- Obama's face isn't detected in my app, only in Camera.app.
- When the camera zooms out from Putin's face, my app draws a rectangle over the right half of his face, cutting it in half, while Camera.app draws a rectangle correctly around his face.
Why is this happening?
My code is as follows. Do you see anything wrong?
First, I create a video output as follows:
let videoOutput = AVCaptureVideoDataOutput() videoOutput.videoSettings = [kCVPixelBufferPixelFormatTypeKey as AnyHashable: Int(kCMPixelFormat_32BGRA)] session.addOutput(videoOutput) videoOutput.setSampleBufferDelegate(faceDetector, queue: faceDetectionQueue)
This is the delegate:
class FaceDetector: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate { func captureOutput(_ captureOutput: AVCaptureOutput!, didOutputSampleBuffer sampleBuffer: CMSampleBuffer!, from connection: AVCaptureConnection!) { let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)! let features = FaceDetector.ciDetector.features( in: CIImage(cvPixelBuffer: imageBuffer)) let faces = features.map { $0.bounds } let imageSize = CVImageBufferGetDisplaySize(imageBuffer) let faceBounds = faces.map { (face: CIFeature) -> CGRect in var ciBounds = face.bounds ciBounds = ciBounds.applying( CGAffineTransform(scaleX: 1/imageSize.width, y: -1/imageSize.height)) CGRect(x: 0, y: 0, width: 1, height: -1).verifyContains(ciBounds) let bounds = ciBounds.applying(CGAffineTransform(translationX: 0, y: 1.0)) CGRect(x: 0, y: 0, width: 1, height: 1).verifyContains(bounds) return bounds } DispatchQueue.main.sync { facesUpdated(faceBounds, imageSize) } } private static let ciDetector = CIDetector(ofType: CIDetectorTypeFace, context: nil, options: [CIDetectorAccuracy: CIDetectorAccuracyHigh])! }
The facesUpdated() callback is as follows:
class PreviewView: UIView { private var faceRects = [UIView]() private static func makeFaceRect() -> UIView { let r = UIView() r.layer.borderWidth = FocusRect.borderWidth r.layer.borderColor = FocusRect.color.cgColor faceRects.append(r) addSubview(r) return r } private func removeAllFaceRects() { for faceRect in faceRects { verify(faceRect.superview == self) faceRect.removeFromSuperview() } faceRects.removeAll() } private func facesUpdated(_ faces: [CGRect], _ imageSize: CGSize) { removeAllFaceRects() let faceFrames = faces.map { (original: CGRect) -> CGRect in let face = original.applying(CGAffineTransform(scaleX: bounds.width, y: bounds.height)) verify(self.bounds.contains(face)) return face } for faceFrame in faceFrames { let faceRect = PreviewView.makeFaceRect() faceRect.frame = faceFrame } } }
I also tried the following, but they didn't help:
- Setting the AVCaptureVideoDataOutput's videoSettings to nil.
- Explicitly setting the CIDetector's orientation to portrait. The phone is in portrait for this test, so it shouldn't matter.
- Setting and removing CIDetectorTracking: true
- Setting and removing CIDetectorAccuracy: CIDetectorAccuracyHigh
- Trying to track only one face, by looking only at the first feature detected.
- Replacing CVImageBufferGetDisplaySize() with CVImageBufferGetEncodedSize() — they're anyway same, at 1440 x 1080.