dual photo delivery

I am trying to request dual photo delivery. I have modified the AVCamPhotoFilter project to deliver dual photos when not in depth mode.


I have opted in for dual photo delivery and requested dual photo delivery in the photoSettings. This seems to be working because I am seeing two AVCapturePhoto callbacks


The first callback works properly with photo.sourceDeviceType set to builtInTelephotoCamera

The second callback does not. Instead of photo.sourceDeviceType set to builtInWideAngleCamera it is set to builtInDualCamera. This returns an empty pixel buffer. Please help

Accepted Reply

I have also confirmed this result. If you request an uncompressed pixel format you get an allocation error on the second callback and the pixel buffer is nil. I've only gotten the dual delivery to work if I choose a compressed pixel format (the default). You can use the fileDataRepresnetation to create a UIImage and then access the CGImage's pixel buffer. But the coordinate system is rotated so be careful.


Any idea if Apple is going to fix this or is uncompressed not supported for the dual delivery mode?

Replies

Can someone please share some example code for dual photo delivery?

Did you receive an error with the callback?

Pseudo code:


// After setting up the AVCaptureSession, adding all inputs and outputs, selecting the desired device activeFormat or session preset

if ( photoOutput.dualCameraDualPhotoDeliverySupported ) {

photoOutput.dualCameraDualPhotoDeliveryEnabled = YES;

}



// When requesting a photo capture, request a dual photos (obviously this only works on a DualCamera)

if ( photoOutput.dualCameraDualPhotoDeliveryEnabled ) {

photoSettings.dualCameraDualPhotoDeliveryEnabled = YES;

}

// issue the request



// Expect twice as many callbacks as before. For instance, if doing a bracketed capture, you'll receive n callbacks for each camera.

// Note that dualPhotoDelivery is only supported via the AVCapturePhoto callback, so your AVCapturePhotoCaptureDelegate must implement the following callback:


- (void)captureOutput:(AVCapturePhotoOutput *)captureOutput didFinishProcessingPhoto:(nonnull AVCapturePhoto *)photo error:(nullable NSError *)error

I am receiving the following error in the second callback


Optional(__ObjC.AVCaptureDevice.DeviceType(_rawValue: AVCaptureDeviceTypeBuiltInTelephotoCamera))

Optional([6765.77 0.00 0.00 | 0.00 6765.77 0.00 | 1997.83 1505.66 1.00], pixelSize:0.001 mm, distortionCenter:{2003.96,1515.25} ref:{4032x3024})

Optional(__ObjC.AVCaptureDevice.DeviceType(_rawValue: AVCaptureDeviceTypeBuiltInDualCamera))

nil

Error occurred while capturing photo: Missing pixel buffer (Optional(Error Domain=AVFoundationErrorDomain Code=-11801 "Cannot Complete Action" UserInfo={NSLocalizedRecoverySuggestion=Try again later., NSLocalizedDescription=Cannot Complete Action, NSUnderlyingError=0x1cc054e50 {Error Domain=NSOSStatusErrorDomain Code=-12786 "(null)"}}))


Shouldn't the device type in the second callback be the wideAngleCamera, not the dualCamera? I think this is the reason for the missing pixel buffer.


I've attached some code snippets below.


// Add photo output

if session.canAddOutput(photoOutput) {

session.addOutput(photoOutput)

photoOutput.isHighResolutionCaptureEnabled = true

if depthVisualizationEnabled {

if photoOutput.isDepthDataDeliverySupported {

photoOutput.isDepthDataDeliveryEnabled = true

} else {

depthVisualizationEnabled = false

}

} else { // depth not enabled, enable dual photo delivery on compatible devices - JB

if photoOutput.isDualCameraDualPhotoDeliverySupported {

photoOutput.isDualCameraDualPhotoDeliveryEnabled = true

} else {

photoOutput.isDualCameraDualPhotoDeliveryEnabled = false

}

}

} else {

print("Could not add photo output to the session")

setupResult = .configurationFailed

session.commitConfiguration()

return

}






@IBAction private func capturePhoto(_ photoButton: UIButton) {

let depthEnabled = depthVisualizationSwitch.isOn

sessionQueue.async {

let photoSettings = AVCapturePhotoSettings(format: [kCVPixelBufferPixelFormatTypeKey as String: Int(kCVPixelFormatType_32BGRA)])

if depthEnabled && self.photoOutput.isDepthDataDeliverySupported {

photoSettings.isDepthDataDeliveryEnabled = true

photoSettings.embedsDepthDataInPhoto = false

}

// Enable dual photo delivery and disable optical image stabilization - JB

if self.photoOutput.isDualCameraDualPhotoDeliverySupported {

photoSettings.isDualCameraDualPhotoDeliveryEnabled = true

photoSettings.isAutoStillImageStabilizationEnabled = false

}

// Enable delivery of camera extrinsics - JB

if self.photoOutput.isCameraCalibrationDataDeliverySupported {

photoSettings.isCameraCalibrationDataDeliveryEnabled = true

}

self.photoOutput.capturePhoto(with: photoSettings, delegate: self)

}

}




// MARK: - Photo Output Delegate

func photoOutput(_ output: AVCapturePhotoOutput, didFinishProcessingPhoto photo: AVCapturePhoto, error: Error?) {

// first callback good - telephoto...2nd callback reads dualCamera instead of wideAngle. Missing pixel buffer

print(photo.sourceDeviceType)

print(photo.cameraCalibrationData)

guard let photoPixelBuffer = photo.pixelBuffer else

print("Error occurred while capturing photo: Missing pixel buffer (\(String(describing: error)))")

return

}

......

-12786 indicates that an allocation is failing. Could you please try setting the AVCapturePhotoSettings output format to 420f instead of BGRA and see if the problem still occurs? (You should file a bug at bugreport.apple.com either way).

No luck. I still get the same error using 420f.


Do you have any projects that implement dual photo delivery that you could share? I remember seeing it demoed at WWDC, but that code was not made available. Thanks!

I have also confirmed this result. If you request an uncompressed pixel format you get an allocation error on the second callback and the pixel buffer is nil. I've only gotten the dual delivery to work if I choose a compressed pixel format (the default). You can use the fileDataRepresnetation to create a UIImage and then access the CGImage's pixel buffer. But the coordinate system is rotated so be careful.


Any idea if Apple is going to fix this or is uncompressed not supported for the dual delivery mode?

Could you please file a bug report at bugreport.apple.com with a small test app that reproduces the problem? Is this on iPhone 7 Plus that you’re seeing the failure? It may be configuration specific.

I am using 8 Plus

Submitted bug 34943112

And