What's new in Camera Capture on iPhone 7 and iPhone 7 Plus

What’s new in Camera Capture on iPhone 7 and iPhone 7 Plus


On September 7, 2016, Apple announced a new generation of iPhones — the 7 and 7 Plus, which feature some remarkable advances in mobile phone camera technology. The following tech note discusses new camera features and how they impact AVFoundation’s capture APIs. The information outlined here is discussed in greater detail at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html.

General Camera Specifications

The iPhone 7 / 7 Plus’s rear-facing wide angle camera has a 28mm focal length (in 35mm film) with a 6-element lens and a larger f/1.8 aperture compared to the iPhone 6s’s f/2.2. The camera’s aperture is discoverable using the -[AVCaptureDevice lensAperture] property introduced in iOS 8.

OIS

Optical image stabilization (OIS) optically centers the camera lens to minimize the effects of hand shake, which is particularly helpful in low light situations where longer exposures are needed, and in video recording. The rear-facing camera on last year’s iPhone 6s Plus supports OIS, whereas the smaller iPhone 6s and iPhone SE only support software-based image stabilization using multi-image fusion. iPhone 7 and 7 Plus’s wide angle cameras both support OIS. These cameras use a combination of optical stabilization and software multi-image fusion to deliver the best possible still images in low light. To capture photos and video with OIS, use the OIS specific APIs detailed at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW4.

Improved Flash

iPhone 7 and 7 Plus have a Quad-LED True Tone flash compared to the 2 LED flash on iPhone 6s / 6s Plus. Embedded in the LEDs is a sensor that detects the flicker frequency of indoor lights and compensates for it in videos and photos. The 4 LEDs also produce a 50% brighter flashlight in Control Center.

Wide Color

Like the previously announced iPad Pro 9.7”, iPhone 7 and iPhone 7 Plus feature wide gamut color displays and cameras capable of capturing still images in the Apple Wide Color (P3 D65) format. Please review 2016 WWDC Session 501 “Advances in iOS Photography” for an overview of wide color capture in iOS 10 ( https://developer.apple.com/videos/play/wwdc2016/501 ). All information pertaining to wide color support in iPad Pro 9.7” also applies to the cameras on iPhone 7 and iPhone 7 Plus. Wide color capture APIs are detailed at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW14.

RAW Capture

As previously announced at WWDC, iOS 10 supports Bayer RAW photo capture on iPhone 6s / 6s Plus, iPhone SE, and iPad Pro 9.7. Add to that list iPhone 7 and 7 Plus. RAW photo capture is currently only supported when using AVCaptureSessionPresetPhoto. For more details, see https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW31.


7 Megapixel Selfies

The front-facing camera on iPhone 6s / 6s Plus supports high resolution stills at 5 megapixels (2576x1932) but only captures video at a maximum resolution of 1280x960. The front-facing camera on iPhone 7 / 7 Plus supports up to 7 megapixel images (3088x2320) and can also stream 7 megapixel images through AVCaptureVideoDataOutput using -[AVCaptureDevice setActiveFormat:]. For video recording, the front facing camera captures a maximum resolution of 1080p30, compared to the iPhone 6s’ maximum resolution of 720p30.


iPhone 7 Plus Dual Cameras

iPhone 7 Plus features two 12 megapixel cameras on the back: one wide-angle (28mm, f/1.8) and one telephoto (56 mm, f/2.8). These two cameras can be discovered and used independently, or as a single virtual camera that automatically switches between physical cameras for zoom — in effect turning two prime lens cameras into a single zoom lens camera. With a proliferation of cameras on iPhone 7 Plus and the need to differentiate between them, AVCaptureDevice now exposes a readonly -deviceType property, which can be one of the following enumerated values:

AVCaptureDeviceTypeBuiltInMicrophone - an audio capture device (microphone)

AVCaptureDeviceTypeBuiltInWideAngleCamera - A wide angle camera. All iOS 10 supported devices present front and back cameras as built-in wide angle cameras (except for the new telephoto camera on iPhone 7 Plus).

AVCaptureDeviceTypeBuiltInTelephotoCamera - The new 56 mm telephoto rear-facing camera on iPhone 7 Plus.

AVCaptureDeviceTypeBuiltInDuoCamera - A virtual camera on iPhone 7 Plus that switches between wide-angle and telephoto seamlessly for a smooth zoom.


The array of AVCaptureDeviceFormats supported by the wide-angle and telephoto camera differ only in their -fieldOfView, -minISO, and -maxISO values. They support the same resolutions and frame rates. Formats are presented in the same order too, so for instance, the ‘420v’ 720p240 format is at index 12 in the -[AVCaptureDevice formats] array on both wide-angle and telephoto cameras. The Dual camera supports the same set of AVCaptureDeviceFormats as the wide-angle camera, except for the slo-mo (120 and 240 fps) formats which are absent. Tables of each of the iPhone 7 Plus cameras' formats is included at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW21.


The Dual camera's defining feature is its ability to smoothly transition between wide and tele cameras, acting like a single lens camera with optical zoom at 2x. Zoom operations are performed on the Dual camera using the familiar -[AVCaptureDevice setVideoZoomFactor:] or -[AVCaptureDevice rampToVideoZoomFactor:withRate:] APIs. Video zoom factor is always expressed in terms of the wide-angle camera, even when only the telephoto camera is in use. When zoomed, the Dual camera intelligently fuses images from the wide-angle and telephoto cameras to improve image quality. This process is transparent to the user and happens automatically when you take pictures using AVCapturePhotoOutput or AVCaptureStillImageOutput. The point at which the cross over from wide-angle to telephoto happens depends on a variety of factors including current focus position, current zoom factor, and current exposure. Because the Dual camera can change at unpredictable times between formats with different ISO ranges and focal lengths, certain AVCaptureDevice manual control APIs are not supported, as the preservation of locked or custom control values would result in visually jarring jumps in focus position, exposure, or white balance when changing between cameras. When using the Dual camera AVCaptureDevice, the following manual control API restrictions apply:

-[AVCaptureDevice setFocusModeLockedWithLensPosition:completionHandler:] throws an exception if set to a position other than AVCaptureLensPositionCurrent.

-[AVCaptureDevice setExposureModeCustomWithDuration:ISO:completionHandler:] always throws an exception, as AVCaptureExposureModeCustom is unsupported.

-[AVCaptureDevice setExposureMode: AVCaptureExposureModeCustom ] always throws an exception, as AVCaptureExposureModeCustom is unsupported.

-[AVCaptureDevice setWhiteBalanceModeLockedWithDeviceWhiteBalanceGains:completionHandler:] throws an exception if set to gains other than AVCaptureWhiteBalanceGainsCurrent.


We’ve provided a new AVCaptureDevice API to allow you to write device agnostic and future-proof code when dealing with the above restrictions.

-[AVCaptureDevice isLockingFocusWithCustomLensPositionSupported] answers NO on the Dual camera, YES on all other focus-capable cameras.

-[AVCaptureDevice isLockingWhiteBalanceWithCustomDeviceGainsSupported] answers NO on the Dual camera, YES on all other cameras.

-[AVCaptureDevice isExposureModeSupported:AVCaptureExposureModeCustom] This API has been available since iOS 4. It answers NO on the Dual camera, YES on all other cameras.


Note that RAW capture is also not supported on the Dual camera. For greater detail about the Dual camera, see https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW32.

AVCaptureDevice Discovery changes in iOS 10

Some shipping apps in the App Store make assumptions about the number and order of AVCaptureDevices presented in +[AVCaptureDevice devices] and +[AVCaptureDevice devicesWithMediaType:]. For instance, many apps assume that the first AVCaptureDevice in the -devices array is the back-facing, wide-angle camera, and the second is the front-facing wide-angle camera. Already-shipping apps that mistakenly pick the Dual camera are at high risk of crashing due to exceptions being thrown by the AVFoundation framework, since the Dual camera has manual control usage restrictions as outlined above. In iOS 10, we’re introducing a new, safer means of discovering AVCaptureDevices called AVCaptureDeviceDiscoverySession, and deprecating +[AVCaptureDevice devices] and +[AVCaptureDevice devicesWithMediaType:]. These latter two APIs continue to work as before, but on iPhone 7 Plus, the telephoto and Dual camera are never published through these deprecated APIs.


AVCaptureDeviceDiscoverySession allows you to search for AVCaptureDevices by deviceType, mediaType, and position. You create a AVCaptureDeviceDiscoverySession instance by passing it an array of deviceTypes for which you’d like to search, and optionally, a media type or position. Note that you can only search for devices of known deviceTypes, so if, in the future, a new deviceType is added to AVCaptureDevice.h, your code remains insulated, and use of the new feature will require an opt-in on your part. Once you’ve created an AVCaptureDeviceDiscoverySession, you can query its -devices property to get its list of AVCaptureDevices that match your search criteria. Values contained in the -devices array are not sorted by any particular search criterion. If you wish to order the results, you must do it with your own sort function. For instance, if you want to search for all known camera types and sort them such that back cameras are before front cameras, presented Dual camera first, then wide-angle, then telephoto, your function might look like so:


static NSArray<AVCaptureDevice *> *allVideoCameras( void )
{
    NSArray *allCameraDeviceTypes = @[AVCaptureDeviceTypeBuiltInDuoCamera, AVCaptureDeviceTypeBuiltInWideAngleCamera, AVCaptureDeviceTypeBuiltInTelephotoCamera ];
    AVCaptureDeviceDiscoverySession *discoverySession = [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:allCameraDeviceTypes mediaType:AVMediaTypeVideo position:AVCaptureDevicePositionUnspecified];
    NSArray<AVCaptureDevice *> *devices = discoverySession.devices;
    return [devices sortedArrayUsingComparator:^NSComparisonResult(AVCaptureDevice * _Nonnull device1, AVCaptureDevice * _Nonnull device2) {
        if ( device1.position != device2.position ) {
            return ( (device1.position < device2.position) ? NSOrderedAscending : NSOrderedDescending );
        }
        NSNumber *device1Index = @([allCameraDeviceTypes indexOfObject:device1.deviceType]);
        NSNumber *device2Index = @([allCameraDeviceTypes indexOfObject:device2.deviceType]);
        return [device1Index compare:device2Index];
    }];
}


As a convenience, we’ve added a new class method to AVCaptureDevice that allows you to access the default device matching the above three search criteria (deviceType, mediaType, position). For instance, if you want to choose the Dual camera if present and fall back to the wide-angle camera, your function might look like so:


static AVCaptureDevice *defaultCameraWithPosition(AVCaptureDevicePosition position)
{
    AVCaptureDevice *defaultCamera = [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInDuoCamera mediaType:AVMediaTypeVideo position:position];
    if ( ! defaultCamera ) {
        defaultCamera = [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInWideAngleCamera mediaType:AVMediaTypeVideo position:position]
    }
    return defaultCamera;
}

AVCaptureDeviceDiscoverySession documentation is available at https://developer.apple.com/reference/avfoundation/avcapturedevicediscoverysession.

Shallow depth of field effect

At the September 7 event, a new Apple Camera app “Portrait” mode was demoed. In this mode, a shallow depth of field effect is applied in real-time to the preview and rendered to still images. This effect leaves the subject sharp, while applying a blur to the background for a pleasing bokeh effect. This feature will become available in Camera app in an iOS 10 software update. This feature is not presently available via API.

Replies

Thanks very much for this excellent rundown of the new features, especially the future proofing.


I've already have a rough beta up and running on the new Dual cameras using these notes and I'm extremely impressed with the virtual camera.


A few oddities of the past weeks now make much more sense. Thanks heaps.


Andy Grace

Hi bford,


The virtual camera is terrific and I cannot wait for my new device getting shipped out.


I have a question though. Will iOS support openning both front and back cameras simultaneously in a near future?


Thanks,

Very useful thanks.


The tele lens will really improve one of my apps.

"Shallow depth of field effect" This feature is not presently available via API.


So it means Apple will not allow developer to use iPhone 7 plus camera to develop camera app with depth of field effect feature. It will disappoint us.

Does anyone know if there is a way to enable/disable the OIS in 10.1 beta or future plans? There are some gimbals out there, specifically the DJI Osmo Mobile that are having issues with the iPhone 6&7 Plus sizes and its because of the OIS in these phones. DJI is hoping Apple can put some fixes into the iOS software that will enable users to disable and enable the OIS in the camera.


Can this be done?

From the cited developer documentation:

For video capture, the

AVCaptureConnection
preferredVideoStabilizationMode
property automatically enables both optical and digital stabilization on devices supporting video OIS.


So if you set the preferredVideoStabilizationMode to off (the default), you will not get any OIS.


Apple's own camera app always opts in for video stabilization, but DJI is welcome to write an app that does not use stabilization and rely upon their own.

Could we 1) capture image and 2) video using the telephoto and wide-angle cameras at the same time possibly in the future release through api?

When can we use Shallow depth of field effect Via API ?😉

Thanks for the summary, I have some additional questions:


1) does the tele camera support OIS like the wide angle camera?

2) is there a stereo camera calibration available for the dual device in the API (intrinsics/extrinsics)?

3) if I capture video from the two cameras independently, will they be synchronized/based on the same system clock?


Thanks

1) Tele does not have OIS. It does support SIS, which is our multi-image fusion stabilization (non-optical) technique, and therefore when taking still images with AVCapturePhotoOutput and the tele, it does report autoStillImageStabilization being supported.

2) No stereo camera calibration is available for public consumption.

3) When you are using the AVCaptureDeviceTypeBuiltInDualCamera that automatically switches between wide and tele, of course they are synchronized to the same clock. Simultaneous running of the AVCaptureDeviceTypeBuiltInTelephotoCamera and AVCaptureDeviceTypeBuiltInWideAngleCamera cameras is not supported, so question 3 is moot. Both cameras are tied to a common clock though.

Wait a minute, could you confirm the last part? "Simultaneous running of the AVCaptureDeviceTypeBuiltInTelephotoCamera and AVCaptureDeviceTypeBuiltInWideAngleCamera cameras is not supported"


If that is true what's the point of having two cameras? I wrote some code last night to create a AVCaptureSession for each at the same time and it seemed to work (although have to confirm the two recorded videos are different). Fused camera is great and all for basic use, but advanced use would require to be able to configure and run both cameras independently.


If it is really not possible, will that possibly be fixed by an API update?

Alright, cannot really work with two sessions actually (if you start one the other stops) and trying to add multiple inputs I get:


[AVCaptureSession addInputWithNoConnections:] Multiple audio/video AVCaptureInputs are not currently supported


So it seems like I cannot use both cameras at the same time. Gutted I took that for granted. I guess even simultaneous independent photos are off the table?

@bford I see in the native camera app you are able to manually adjust exposure with both 1x and 2x options. What is the native camera app doing for exposure adjustment even with telephoto camera ? Also what is the default camera used by native camera app - dual camera or wide angle camera ?

Apple's Camera app uses the DualCamera (auto-switching) camera. That camera doesn't support manual exposure controls (setting a custom exposureDuration / ISO), but it does support setting an exposureBias, which is what camera uses when bumping exposure brighter or darker.

Is the new AVCaptureDeviceTypeBuiltInDuoCamera available on iOS 10.0 device? I have met crashes when using the

[AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInDuoCamera mediaType:AVMediaTypeVideo position:position];

API on iOS 10.0 devices, but it works perfectly in iOS 10.1.1 devices, is there any changes made in iOS 10.1?