18 Replies
      Latest reply on Jul 27, 2017 2:07 PM by sdbunger
      bford Apple Staff Apple Staff (315 points)

        What’s new in Camera Capture on iPhone 7 and iPhone 7 Plus

         

        On September 7, 2016, Apple announced a new generation of iPhones — the 7 and 7 Plus, which feature some remarkable advances in mobile phone camera technology. The following tech note discusses new camera features and how they impact AVFoundation’s capture APIs. The information outlined here is discussed in greater detail at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html.


        General Camera Specifications

        The iPhone 7 / 7 Plus’s rear-facing wide angle camera has a 28mm focal length (in 35mm film) with a 6-element lens and a larger f/1.8 aperture compared to the iPhone 6s’s f/2.2. The camera’s aperture is discoverable using the -[AVCaptureDevice lensAperture] property introduced in iOS 8.


        OIS

        Optical image stabilization (OIS) optically centers the camera lens to minimize the effects of hand shake, which is particularly helpful in low light situations where longer exposures are needed, and in video recording. The rear-facing camera on last year’s iPhone 6s Plus supports OIS, whereas the smaller iPhone 6s and iPhone SE only support software-based image stabilization using multi-image fusion. iPhone 7 and 7 Plus’s wide angle cameras both support OIS. These cameras use a combination of optical stabilization and software multi-image fusion to deliver the best possible still images in low light. To capture photos and video with OIS, use the OIS specific APIs detailed at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW4.


        Improved Flash

        iPhone 7 and 7 Plus have a Quad-LED True Tone flash compared to the 2 LED flash on iPhone 6s / 6s Plus. Embedded in the LEDs is a sensor that detects the flicker frequency of indoor lights and compensates for it in videos and photos. The 4 LEDs also produce a 50% brighter flashlight in Control Center.


        Wide Color

        Like the previously announced iPad Pro 9.7”, iPhone 7 and iPhone 7 Plus feature wide gamut color displays and cameras capable of capturing still images in the Apple Wide Color (P3 D65) format. Please review 2016 WWDC Session 501 “Advances in iOS Photography” for an overview of wide color capture in iOS 10 ( https://developer.apple.com/videos/play/wwdc2016/501 ). All information pertaining to wide color support in iPad Pro 9.7” also applies to the cameras on iPhone 7 and iPhone 7 Plus. Wide color capture APIs are detailed at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW14.


        RAW Capture

        As previously announced at WWDC, iOS 10 supports Bayer RAW photo capture on iPhone 6s / 6s Plus, iPhone SE, and iPad Pro 9.7. Add to that list iPhone 7 and 7 Plus. RAW photo capture is currently only supported when using AVCaptureSessionPresetPhoto. For more details, see https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW31.

         

        7 Megapixel Selfies

        The front-facing camera on iPhone 6s / 6s Plus supports high resolution stills at 5 megapixels (2576x1932) but only captures video at a maximum resolution of 1280x960. The front-facing camera on iPhone 7 / 7 Plus supports up to 7 megapixel images (3088x2320) and can also stream 7 megapixel images through AVCaptureVideoDataOutput using -[AVCaptureDevice setActiveFormat:]. For video recording, the front facing camera captures a maximum resolution of 1080p30, compared to the iPhone 6s’ maximum resolution of 720p30.

         

        iPhone 7 Plus Dual Cameras

        iPhone 7 Plus features two 12 megapixel cameras on the back: one wide-angle (28mm, f/1.8) and one telephoto (56 mm, f/2.8). These two cameras can be discovered and used independently, or as a single virtual camera that automatically switches between physical cameras for zoom — in effect turning two prime lens cameras into a single zoom lens camera. With a proliferation of cameras on iPhone 7 Plus and the need to differentiate between them, AVCaptureDevice now exposes a readonly -deviceType property, which can be one of the following enumerated values:

             AVCaptureDeviceTypeBuiltInMicrophone - an audio capture device (microphone)

             AVCaptureDeviceTypeBuiltInWideAngleCamera - A wide angle camera. All iOS 10 supported devices present front and back cameras as built-in wide angle cameras (except for the new telephoto camera on iPhone 7 Plus).

             AVCaptureDeviceTypeBuiltInTelephotoCamera - The new 56 mm telephoto rear-facing camera on iPhone 7 Plus.

             AVCaptureDeviceTypeBuiltInDuoCamera - A virtual camera on iPhone 7 Plus that switches between wide-angle and telephoto seamlessly for a smooth zoom.

         

        The array of AVCaptureDeviceFormats supported by the wide-angle and telephoto camera differ only in their -fieldOfView, -minISO, and -maxISO values. They support the same resolutions and frame rates. Formats are presented in the same order too, so for instance, the ‘420v’ 720p240 format is at index 12 in the -[AVCaptureDevice formats] array on both wide-angle and telephoto cameras. The Dual camera supports the same set of AVCaptureDeviceFormats as the wide-angle camera, except for the slo-mo (120 and 240 fps) formats which are absent. Tables of each of the iPhone 7 Plus cameras' formats is included at https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW21.

         

        The Dual camera's defining feature is its ability to smoothly transition between wide and tele cameras, acting like a single lens camera with optical zoom at 2x. Zoom operations are performed on the Dual camera using the familiar -[AVCaptureDevice setVideoZoomFactor:] or -[AVCaptureDevice rampToVideoZoomFactor:withRate:] APIs. Video zoom factor is always expressed in terms of the wide-angle camera, even when only the telephoto camera is in use. When zoomed, the Dual camera intelligently fuses images from the wide-angle and telephoto cameras to improve image quality. This process is transparent to the user and happens automatically when you take pictures using AVCapturePhotoOutput or AVCaptureStillImageOutput. The point at which the cross over from wide-angle to telephoto happens depends on a variety of factors including current focus position, current zoom factor, and current exposure. Because the Dual camera can change at unpredictable times between formats with different ISO ranges and focal lengths, certain AVCaptureDevice manual control APIs are not supported, as the preservation of locked or custom control values would result in visually jarring jumps in focus position, exposure, or white balance when changing between cameras. When using the Dual camera AVCaptureDevice, the following manual control API restrictions apply:


             -[AVCaptureDevice setFocusModeLockedWithLensPosition:completionHandler:] throws an exception if set to a position other than AVCaptureLensPositionCurrent.

             -[AVCaptureDevice setExposureModeCustomWithDuration:ISO:completionHandler:] always throws an exception, as AVCaptureExposureModeCustom is unsupported.

             -[AVCaptureDevice setExposureMode: AVCaptureExposureModeCustom ] always throws an exception, as AVCaptureExposureModeCustom is unsupported.

             -[AVCaptureDevice setWhiteBalanceModeLockedWithDeviceWhiteBalanceGains:completionHandler:] throws an exception if set to gains other than AVCaptureWhiteBalanceGainsCurrent.

         

        We’ve provided a new AVCaptureDevice API to allow you to write device agnostic and future-proof code when dealing with the above restrictions.

             -[AVCaptureDevice isLockingFocusWithCustomLensPositionSupported] answers NO on the Dual camera, YES on all other focus-capable cameras.

             -[AVCaptureDevice isLockingWhiteBalanceWithCustomDeviceGainsSupported] answers NO on the Dual camera, YES on all other cameras.

             -[AVCaptureDevice isExposureModeSupported:AVCaptureExposureModeCustom] This API has been available since iOS 4. It answers NO on the Dual camera, YES on all other cameras.

         

        Note that RAW capture is also not supported on the Dual camera. For greater detail about the Dual camera, see https://developer.apple.com/library/content/documentation/DeviceInformation/Reference/iOSDeviceCompatibility/Cameras/Cameras.html#//apple_ref/doc/uid/TP40013599-CH107-SW32.


        AVCaptureDevice Discovery changes in iOS 10

        Some shipping apps in the App Store make assumptions about the number and order of AVCaptureDevices presented in +[AVCaptureDevice devices] and +[AVCaptureDevice devicesWithMediaType:]. For instance, many apps assume that the first AVCaptureDevice in the -devices array is the back-facing, wide-angle camera, and the second is the front-facing wide-angle camera. Already-shipping apps that mistakenly pick the Dual camera are at high risk of crashing due to exceptions being thrown by the AVFoundation framework, since the Dual camera has manual control usage restrictions as outlined above. In iOS 10, we’re introducing a new, safer means of discovering AVCaptureDevices called AVCaptureDeviceDiscoverySession, and deprecating +[AVCaptureDevice devices] and +[AVCaptureDevice devicesWithMediaType:]. These latter two APIs continue to work as before, but on iPhone 7 Plus, the telephoto and Dual camera are never published through these deprecated APIs.

         

        AVCaptureDeviceDiscoverySession allows you to search for AVCaptureDevices by deviceType, mediaType, and position. You create a AVCaptureDeviceDiscoverySession instance by passing it an array of deviceTypes for which you’d like to search, and optionally, a media type or position. Note that you can only search for devices of known deviceTypes, so if, in the future, a new deviceType is added to AVCaptureDevice.h, your code remains insulated, and use of the new feature will require an opt-in on your part. Once you’ve created an AVCaptureDeviceDiscoverySession, you can query its -devices property to get its list of AVCaptureDevices that match your search criteria. Values contained in the -devices array are not sorted by any particular search criterion. If you wish to order the results, you must do it with your own sort function. For instance, if you want to search for all known camera types and sort them such that back cameras are before front cameras, presented Dual camera first, then wide-angle, then telephoto, your function might look like so:

         

        static NSArray<AVCaptureDevice *> *allVideoCameras( void )
        {
            NSArray *allCameraDeviceTypes = @[AVCaptureDeviceTypeBuiltInDuoCamera, AVCaptureDeviceTypeBuiltInWideAngleCamera, AVCaptureDeviceTypeBuiltInTelephotoCamera ];
            AVCaptureDeviceDiscoverySession *discoverySession = [AVCaptureDeviceDiscoverySession discoverySessionWithDeviceTypes:allCameraDeviceTypes mediaType:AVMediaTypeVideo position:AVCaptureDevicePositionUnspecified];
            NSArray<AVCaptureDevice *> *devices = discoverySession.devices;
            return [devices sortedArrayUsingComparator:^NSComparisonResult(AVCaptureDevice * _Nonnull device1, AVCaptureDevice * _Nonnull device2) {
                if ( device1.position != device2.position ) {
                    return ( (device1.position < device2.position) ? NSOrderedAscending : NSOrderedDescending );
                }
                NSNumber *device1Index = @([allCameraDeviceTypes indexOfObject:device1.deviceType]);
                NSNumber *device2Index = @([allCameraDeviceTypes indexOfObject:device2.deviceType]);
                return [device1Index compare:device2Index];
            }];
        }
        

         

        As a convenience, we’ve added a new class method to AVCaptureDevice that allows you to access the default device matching the above three search criteria (deviceType, mediaType, position). For instance, if you want to choose the Dual camera if present and fall back to the wide-angle camera, your function might look like so:

         

        static AVCaptureDevice *defaultCameraWithPosition(AVCaptureDevicePosition position)
        {
            AVCaptureDevice *defaultCamera = [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInDuoCamera mediaType:AVMediaTypeVideo position:position];
            if ( ! defaultCamera ) {
                defaultCamera = [AVCaptureDevice defaultDeviceWithDeviceType:AVCaptureDeviceTypeBuiltInWideAngleCamera mediaType:AVMediaTypeVideo position:position]
            }
            return defaultCamera;
        }
        

        AVCaptureDeviceDiscoverySession documentation is available at https://developer.apple.com/reference/avfoundation/avcapturedevicediscoverysession.


        Shallow depth of field effect

        At the September 7 event, a new Apple Camera app “Portrait” mode was demoed. In this mode, a shallow depth of field effect is applied in real-time to the preview and rendered to still images. This effect leaves the subject sharp, while applying a blur to the background for a pleasing bokeh effect. This feature will become available in Camera app in an iOS 10 software update. This feature is not presently available via API.