iOS 12 .portraitEffectsMatte only works if device in landscape orientation?

I'm testing out the new iOS 12 AVCapturePhotoOutput.portraitEffectsMatte.

When it works, it's really great!

But for some reason the matte doesn't get generated if the phone is standing up in a portrait orientation (the matte is always absent no matter what lighting conditions I tried). In the same conditions, the matte IS generated if I change the orientation of the phone such that it is in landscape orientation.


Is this a known limitation of portraitEffectsMatte, or am I missing something?


thanks for any help

-tim

Replies

Portrait effects matte generation is not dependent on orientation. It _is_ dependent on a person being in the scene. If there are no detected faces in the frame, the portrait effects matte will not be generated. This high res foreground background segmentation is currently trained on people, so if there are no people in the frame, it cannot be generated.

Hi bford,

There were always people in the frame.

The portrait matte is simply not generated in potrait orientation even if there are people present, and the exact same people present in landscape orientation makes it work.

One thing I had an idea about: I had hard-set the AVCaptureConnection.videoOrientation to .landscapeRight. Perhaps the portraitMatte generation requires the image to be in .up configuration to find the faces etc? To test this i've started to set .videoOrientation as the device rotates.

I don't have the device anymore to try this out, but I will soon.

There’s not much documentation on this. I’m able to save a portrait photo with a matte, but the matte is in a different orientation than the image. The odd thing is that the portrait image works with Apple‘s own camera App, so it must adjust the Matt to match the image.


I’ve been trying to figure out how to use the AVCapturePhotoFileDataRepresentationCustomizer but no luck.

Closing the loop here. Thanks for reporting the issue. A bug was discovered in the -[AVPortraitEffectsMatte portraitEffectsMatteByApplyingExifOrientation:] when passing the native orientation of the buffer. So on the DualCamera, if you request a portraitEffectsMatteByApplyingExifOrientation with the equivalent of .landscapeRight, it will give you a nil buffer.


The workaround is to not try to make a derivative portrait effects matte when no rotation / mirroring is necessary.

Portrait photos taken with the default camera have orientation matching images and mattes. But when taken with the APIs like below, the matte is rotated 90 degrees left of the image. So if we cannot currently use ApplyExifOrientation, how do we take photos that require no rotation like the default camera?


From didFinishProcessingPhoto


PHPhotoLibrary.requestAuthorization { status in

guard status == .authorized else { return }

PHPhotoLibrary.shared().performChanges(

{

let creationRequest = PHAssetCreationRequest.forAsset()

let options = PHAssetResourceCreationOptions()

//options.shouldMoveFile = true

creationRequest.addResource(with: .photo, data: photo.fileDataRepresentation()!, options:options)

}, completionHandler: nil)

}


}

Something's getting lost in translation. Could you please file a bug at bugreport.apple.com, and include a small test xcode project that reproduces the problem?

Ok, submitted bug report.