0 Replies
      Latest reply on Mar 28, 2019 5:46 PM by kart
      kart Level 1 Level 1 (0 points)
        I'm trying to detect faces in my iOS camera app, but it doesn't work properly, while it works properly in Camera.app. Notice that:
        • The first face isns't detected in my app, only in Camera.app.
        • For the third face — the east asian woman — Camera.app correctly draws a rectangle around her face, while my app draws a rectangle that extends far below her face.
        • Obama's face isn't detected in my app, only in Camera.app.
        • When the camera zooms out from Putin's face, my app draws a rectangle over the right half of his face, cutting it in half, while Camera.app draws a rectangle correctly around his face.


        Why is this happening?


        My code is as follows. Do you see anything wrong?


        First, I create a video output as follows:


        let videoOutput = AVCaptureVideoDataOutput()
        videoOutput.videoSettings =
            [kCVPixelBufferPixelFormatTypeKey as AnyHashable:
        videoOutput.setSampleBufferDelegate(faceDetector, queue: faceDetectionQueue)


        This is the delegate:


        class FaceDetector: NSObject, AVCaptureVideoDataOutputSampleBufferDelegate {
          func captureOutput(_ captureOutput: AVCaptureOutput!,
                             didOutputSampleBuffer sampleBuffer: CMSampleBuffer!,
                             from connection: AVCaptureConnection!) {
            let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)!
            let features = FaceDetector.ciDetector.features(
                in: CIImage(cvPixelBuffer: imageBuffer))
            let faces = features.map { $0.bounds }
            let imageSize = CVImageBufferGetDisplaySize(imageBuffer)
            let faceBounds = faces.map { (face: CIFeature) -> CGRect in
                var ciBounds = face.bounds
                ciBounds = ciBounds.applying(
                    CGAffineTransform(scaleX: 1/imageSize.width, y: -1/imageSize.height))
                CGRect(x: 0, y: 0, width: 1, height: -1).verifyContains(ciBounds)
                let bounds = ciBounds.applying(CGAffineTransform(translationX: 0, y: 1.0))
                CGRect(x: 0, y: 0, width: 1, height: 1).verifyContains(bounds)
                return bounds
            DispatchQueue.main.sync {
              facesUpdated(faceBounds, imageSize)
          private static let ciDetector = CIDetector(ofType: CIDetectorTypeFace,
              context: nil,
              options: [CIDetectorAccuracy: CIDetectorAccuracyHigh])!


        The facesUpdated() callback is as follows:


        class PreviewView: UIView {
          private var faceRects = [UIView]()
          private static func makeFaceRect() -> UIView {
            let r = UIView()
            r.layer.borderWidth = FocusRect.borderWidth
            r.layer.borderColor = FocusRect.color.cgColor
            return r
          private func removeAllFaceRects() {
            for faceRect in faceRects {
              verify(faceRect.superview == self)
          private func facesUpdated(_ faces: [CGRect], _ imageSize: CGSize) {
            let faceFrames = faces.map { (original: CGRect) -> CGRect in
                let face = original.applying(CGAffineTransform(scaleX: bounds.width, y: bounds.height))
                return face
            for faceFrame in faceFrames {
              let faceRect = PreviewView.makeFaceRect()
              faceRect.frame = faceFrame


        I also tried the following, but they didn't help:


        • Setting the AVCaptureVideoDataOutput's videoSettings to nil.
        • Explicitly setting the CIDetector's orientation to portrait. The phone is in portrait for this test, so it shouldn't matter.
        • Setting and removing CIDetectorTracking: true
        • Setting and removing CIDetectorAccuracy: CIDetectorAccuracyHigh
        • Trying to track only one face, by looking only at the first feature detected.
        • Replacing CVImageBufferGetDisplaySize() with CVImageBufferGetEncodedSize() — they're anyway same, at 1440 x 1080.