What’s new in AV Foundation Capture for iPhone 6s, iPhone 6s Plus, and iOS 9

The newly announced iPhone 6s and 6s Plus support a myriad of new camera features and video formats. Here’s an overview of what’s new and how to access it using the AV Foundation capture APIs.


12 MP Stills

Both iPhone 6s and 6s Plus can capture 12 megapixel photos (4032x3024) on the rear-facing camera via the AVCaptureStillImageOutput, and can deliver up to 30 fps 12 MP frames to your process via AVCaptureVideoDataOutput. When you use AVCaptureSessionPresetPhoto as your AVCaptureSession’s -sessionPreset, the 12 megapixel ‘420f’ format is chosen by default. You can also access the 12 MP formats by querying the rear-facing camera AVCaptureDevice’s -formats property, and select the one you want using AVCaptureDevice’s -setActiveFormat: API. Note that in addition to 12 MP, the rear-facing camera also supports the photo resolutions seen on earlier iPhones, 8 MP (3264x2448) and 5 MP (2592x1936) in both ‘420f’ and ‘420v’ flavors. These smaller 4:3 photo formats are only accessible using the AVCaptureDevice -setActiveFormat: API.



4K & 1080p120 Video

Both iPhone 6s and 6s plus can shoot 4K (3840x2160) video at up to 30 fps. On previous iPhones, the AVCaptureSessionPresetHigh preset has always been equated with the largest supported 16:9 video resolution. Not so on iPhone 6s and 6s Plus, where the default High preset produces 1080p30 video. 4K video consumes roughly 375 MB of storage per minute compared to 1080p30’s 130 MB per minute. The 4K video formats can be found in the rear-facing camera’s -formats array, and can be selected using AVCaptureDevice’s -setActiveFormat API, or by setting the AVCaptureSession’s -sessionPreset to AVCaptureSessionPreset3840x2160. In addition to 4K30, iPhone 6s and 6s Plus expose separate 720p30, 720p60, 1080p30, and 1080p60 formats. For slo-motion video applications, iPhone 6s and 6s Plus support a new binned 1080p120 format in addition to the 720p240 format first seen on iPhone 6 and 6 Plus. All high-frame rate formats (60 fps and higher) are only accessible using the AVCaptureDevice -setActiveFormat: API.

5 MP Selfies

iPhone 6s and 6s Plus front-facing cameras allow you to take 5 MP still images, but not stream continuously at 5 MP for video data output or movie recording. Beginning with iOS 8 and the iPhone 6 and 6 Plus, the AVCaptureStillImageOutput supports delivery of still images that are larger than the streaming resolution of the camera. Each camera publishes an array of AVCaptureDeviceFormats that have a streaming width and height (attainable by calling CMVideoFormatDescriptionGetDimensions() on the AVCaptureDeviceFormat’s -formatDescription), and a separate high resolution stills width and height (see AVCaptureDeviceFormat’s highResolutionStillImageDimensions property). If you NSLog the array of formats returned when you query the front facing camera for its -formats property, you'll see that the last format in the list has a streaming resolution of 1280x960, and a high resolution still image format of 2576x1932:
[13]<AVCaptureDeviceFormat: 'vide'/'420f' 1280x 960, { 2- 60 fps}, HRSI:2576x1932, fov:54.400, max zoom:60.00 (upscales @1.00), ISO:34.0-2176.0, SS:0.000013-0.500000>

In order to take 5 MP stills from the front-facing camera, you must set your AVCaptureSession’s sessionPreset to AVCaptureSessionPresetPhoto (or select the above format using AVCaptureDevice -setActiveFormat: API) and opt into high-res images using AVCaptureStillImageOutput's highResolutionStillImageOutputEnabled property. Setting this property may cause some rebuilding of the capture render pipeline, so it will result in a glitch in video preview. Therefore, it's better to opt in for high resolution stills before you call AVCaptureSession -startRunning.

Retina Flash
iPhone 6s and 6s Plus contain a custom display chip that allows the retina display to briefly flash 3 times brighter than its usual maximum illuminance. No new API was added to support this feature. Since iOS 4, AVCaptureDevice has supported the -hasFlash, -isFlashModeSupported: and -flashMode properties. The iPhone 6s and 6s Plus front-facing cameras are the first front-facing iOS cameras to respond YES to the -hasFlash property. By setting the front-facing camera's flashMode to AVCaptureFlashModeOn or AVCaptureFlashModeAuto, the retina flash fires when a still image is captured (see AVCaptureStillImageOutput’s captureStillImageAsynchronouslyFromConnection:completionHandler:), just as the True Tone flash fires for rear-facing camera stills.


Live Photos
On iPhone 6s and 6s Plus, Apple’s Camera app captures still images that spring to life when 3D-touched. The moments just before and after the still are captured with motion and sound. AV Foundation does not support Live Photos capture in iOS 9, though LivePhotos can be accessed for playback and sharing using the Photos framework in iOS 9.1, see https://developer.apple.com/library/prerelease/ios/releasenotes/General/WhatsNewIniOS/Articles/iOS9_1.html. Live Photos may be shared with and viewed on any iOS device running iOS 9, or Mac OS X machine running El Capitan.


Optical Image Stabilization During Bracketed Capture

The rear-facing cameras on iPhone 6s Plus and iPhone 6 Plus support optical image stabilization (OIS). When you take low-light pictures using AVCaptureStillImageOutput and automaticallyEnablesStillImageStabilizationWhenAvailable is set to YES, the lens is moved during the capture to counter user hand-shake. The optically stabilized image is then further digitally stabilized to minimize motion blur and recover detail. In iOS 8, this was the only API available to engage optical image stabilization. New in iOS 9, optical image stabilization can be enabled separately from digital image stabilization when taking bracketed still image captures. First, opt in for OIS by setting AVCaptureStillImageOutput’s lensStabilizationDuringBracketedCaptureEnabled to YES, then use captureStillImageBracketAsynchronouslyFromConnection:withSettingsArray:completionHandler: to take an auto or manual exposure bracket. If you wish to take a single still image with OIS, you may call captureStillImageBracketAsynchronouslyFromConnection:withSettingsArray:completionHandler: with an array of 1. Note that OIS is more effective with longer-exposure captures, and offers limited benefit for exposure durations shorter than 1/30th of a second. The lens stabilization module has a limited correction range and so, depending on the amount of shake and exposure duration, may run out of range on one or more image in the bracket. You can tell if lens stabilization was active for a given frame in the bracket by checking each emitted CMSampleBuffer in the bracket for the kCMSampleBufferAttachmentKey_StillImageLensStabilizationInfo attachment (see CMGetAttachment).


Optical Image Stabilization During Video

On iPhone 6 Plus, optical image stabilization is only used to improve low light still image captures. On iPhone 6s Plus, optical image stabilization is used for low light still image captures as well as for stabilizing video for video data output or movie file output. All the rear-facing camera non-slow motion (60 fps or less) 16:9 video formats are optically stabilized on iPhone 6s Plus if you set AVCaptureConnection’s -preferredVideoStabilizationMode to something other than AVCaptureVideoStabilizationModeOff.



Expanded Cinematic Video Stabilization Support
In iOS 8, iPhone 6 and 6 Plus introduced an enhanced, dramatic video stabilization mode entitled “Cinematic”. Cinematic video stabilization was only supported on the rear-facing camera for 1080p30 and 1080p60 on these products. In iOS 9 we’ve expanded cinematic video stabilization support to 720p30 and 720p60, and supported devices to iPad Mini 4, iPad Pro, 6th gen iPod touch, iPhone 6 and 6 Plus, and iPhone 6s and 6s Plus. Cinematic video stabilization is the default video stabilization mode now (aka AVCaptureVideoStabilizationModeAuto) for 720p30, 720p60, 1080p30, and 1080p60 if your app is linked on or after iOS 9.0. See AVCaptureConnection’s preferredVideoStabilizationMode property.


Expanded Support for Recording Timed Metadata Overview

New in iOS 9, clients can record arbitrary timed metadata, detected face metadata, and video orientation metadata to QuickTime movie files using the AVCaptureMovieFileOutput. Clients have been able to add timed metadata samples to QuickTime movies using AVAssetWriter since iOS 8. AVAssetWriter is the preferred interface to use when a client needs access to video and audio sample buffers prior to writing them to a movie. AVCaptureMovieFileOutput is the simpler means of writing QuickTime movies. It 1) requires less set-up code, 2) requires no CMSampleBuffer handling by the application, and 3) is more efficient, since buffers do not need to be messaged into the client process. When you call -addInput: or -addOutput: on an AVCaptureSession instance, it automatically forms AVCaptureConnections between each compatible input and output. In the case of AVCaptureMovieFileOutput, each connection formed corresponds to a track in the resulting movie file. Because writing timed metadata increases file size and CPU usage, and metadata availability may differ from device to device, AVCaptureSession makes an exception — it does not automatically form connections to an AVCaptureMovieFileOutput for metadata; clients must opt-in for the specific metadata that they wish to capture. The AV Foundation framework provides native support for writing two types of timed metadata — detected faces and video orientation.



Expanded Support for Recording Timed Metadata: Detected Faces

Detected face information has been available to client applications since iOS 6. AVMetadataFaceObjects are provided to clients via an AVCaptureMetadataOutput. Face objects are provided to the output by an AVCaptureDeviceInput's input port that also delivers other types of metadata objects, such as detected barcodes. AVCaptureMovieFileOutput does not support the writing of disparately timed metadata to a common track, therefore in iOS 9, AVCaptureInputs may expose a new type of port that delivers just one kind of metadata. This new input port is identifiable by its mediaType of AVMediaTypeMetadata. Clients invoke the input port’s -formatDescription property to determine what kind of metadata it offers, then manually form an AVCaptureConnection between it and the AVCaptureMovieFileOutput. The following code snippet, borrowed from the recently released AVMetadataRecordPlay sample code, demonstrates how to do this:


- (void)connectSpecificMetadataPort:(NSString *)metadataIdentifier

{

for ( AVCaptureInputPort *inputPort in self.videoDeviceInput.ports ) {

CMFormatDescriptionRef desc = inputPort.formatDescription;

if ( desc && ( kCMMediaType_Metadata == CMFormatDescriptionGetMediaType( desc ) ) ) {

CFArrayRef metadataIdentifiers = CMMetadataFormatDescriptionGetIdentifiers( desc );

if ( [(__bridge NSArray *)metadataIdentifiers containsObject:metadataIdentifier] )

{

AVCaptureConnection *connection = [AVCaptureConnection connectionWithInputPorts:@[inputPort] output:self.movieFileOutput];

[self.session addConnection:connection];

break;

}

}

}

}


As mentioned previously, AVCaptureSession never automatically forms connections between eligible outputs and input ports of type AVMediaTypeMetadata. AVCaptureSession _does_ automatically form connections between eligible outputs and input ports of type AVMediaTypeMetadataObject (the input ports used to deliver metadata to an AVCaptureMetadataOutput). To test the legality of your manually created connection, be sure to call [session canAddConnection:proposedConnection] before attempting to add it to the session. Beware that when removing AVCaptureInputs (such as when changing cameras), all connections are severed, so you must form new manual connections between your new camera input and your AVCaptureMovieFileOutput.

Expanded Support for Recording Timed Metadata: Video Orientation

The second type of timed metadata natively supported by AVCaptureMovieFileOutput is video orientation. Often, users start recording a movie with their iOS device in one orientation and then partway through, rotate the device to a new orientation. A video orientation timed metadata track captures these changes and allows a client application to discover when these changes occurred and make edits or custom playback decisions if desired. The following code snippet from AVMetadataRecordPlay demonstrates how to opt in for video orientation metadata recording in your movie file:


[self.session addOutput:movieFileOutput];

AVCaptureConnection *connection = [movieFileOutput connectionWithMediaType:AVMediaTypeVideo];

[movieFileOutput setRecordsVideoOrientationAndMirroringChanges:YES asMetadataTrackForConnection:connection];


Once you’ve opted in, your changes to the connection’s videoOrientation or videoMirrored properties will be recorded to the QuickTime movie file as metadata samples in a timed metadata track. Each video orientation or mirroring change is represented as an EXIF orientation tag value. Beware that when removing AVCaptureInputs (such as when changing cameras), the -setRecordsVideoOrientationAndMirroringChanges:asMetadataTrackForConnection: property must be set to YES again since a new connection will have been formed between the new camera input and the AVCaptureMovieFileOutput.

Expanded Support for Recording Timed Metadata: Client-Provided Metadata

New for iOS 9 is the AVCaptureMetadataInput class, which offers a conduit for timed metadata between the client application and the AVCaptureMovieFileOutput. With AVCaptureMetadataInput, you can record arbitrary timed metadata to your movie. One common example is location (GPS) data. To make use of this new feature you must:

1. Create and add an AVCaptureMetadataInput. Required for creation is a CMFormatDescription that describes the metadata to be provided, and a CMClock that provides a timing reference. The CMClock is critical in allowing the AVCaptureSession to synchronize the client's metadata stream with the other media being captured. The following code snippet from AVMetadataRecordPlay demonstrates how to create an input for location data:


// Create a format description for the location metadata.

NSArray *specs = @[@{ (__bridge id)kCMMetadataFormatDescriptionMetadataSpecificationKey_Identifier : AVMetadataIdentifierQuickTimeMetadataLocationISO6709,

(__bridge id)kCMMetadataFormatDescriptionMetadataSpecificationKey_DataType : (__bridge id)kCMMetadataDataType_QuickTimeMetadataLocation_ISO6709 }];

CMFormatDescriptionRef locationMetadataDesc = NULL;

CMMetadataFormatDescriptionCreateWithMetadataSpecifications(kCFAllocatorDefault, kCMMetadataFormatType_Boxed, (__bridge CFArrayRef)specs, &locationMetadataDesc);


// Create the metadata input for location metadata.

AVCaptureMetadataInput *newLocationMetadataInput = [[AVCaptureMetadataInput alloc] initWithFormatDescription:locationMetadataDesc clock:CMClockGetHostTimeClock()];

CFRelease( locationMetadataDesc );


[self.session addInputWithNoConnections:newLocationMetadataInput];


2. Create and add a connection between the AVCaptureMetadataInput's sole AVCaptureInputPort and the AVCaptureMovieFileOutput. (*Reminder* - AVCaptureSession does not automatically form connections between input ports of type AVMediaTypeMetadata and AVCaptureMovieFileOutput instances). Example:


AVCaptureInputPort *inputPort = [newLocationMetadataInput.ports firstObject];

[self.session addConnection:[AVCaptureConnection connectionWithInputPorts:@[inputPort] output:self.movieFileOutput]];

3. While the session is running, provide metadata in the form of AVTimedMetadataGroups. An AVTimedMetadataGroup consists of an array of AVMetadataItems and a timestamp (in the context of the CMClock provided on creation). It is important that the AVMetadataItems strictly adhere to the CMFormatDescription provided to the AVCaptureMetadataInput's init method. The following code creates a location metadata item, wraps it in an AVTimedMetadataGroup, and appends it to the AVCaptureMetadataInput:


AVMutableMetadataItem *newLocationMetadataItem = [[AVMutableMetadataItem alloc] init];

newLocationMetadataItem.identifier = AVMetadataIdentifierQuickTimeMetadataLocationISO6709;

newLocationMetadataItem.dataType = (__bridge NSString *)kCMMetadataDataType_QuickTimeMetadataLocation_ISO6709;

...

newLocationMetadataItem.value = iso6709Notation;


AVTimedMetadataGroup *metadataItemGroup = [[AVTimedMetadataGroup alloc] initWithItems:@[newLocationMetadataItem] timeRange:CMTimeRangeMake( CMClockGetTime( CMClockGetHostTimeClock() ), kCMTimeInvalid )];


NSError *error = nil;

if ( ! [self.locationMetadataInput appendTimedMetadataGroup:metadataItemGroup error:&error] ) {

NSLog( @"appendTimedMetadataGroup failed with error %@", error );

}


The AVCaptureSession serializes the AVMetadataItems and creates a sample with the timing information provided by the AVTimedMetadataGroup. The sample is written to the file, and its duration is determined by the addition of a follow-on AVTimedMetadataGroup. If you wish to express a period where there is no valid metadata available, simply supply an empty AVTimedMetadataGroup:


AVTimedMetadataGroup *metadataItemGroupToDeclareNoMetadataIsAvailableStartingAtThisTime = [[AVTimedMetadataGroup alloc] initWithItems:@[] timeRange:CMTimeRangeMake( theTime ), kCMTimeInvalid )];

NSError *error = nil;

if ( ! [self.locationMetadataInput appendTimedMetadataGroup:metadataItemGroupToDeclareNoMetadataIsAvailableStartingAtThisTime error:&error] ) {

NSLog( @"appendTimedMetadataGroup failed with error %@", error );

}


Please refer to the AVMetadataRecordPlay sample code and documentation for more information on metadata recording and retrieval during playback.



Camera Restrictions in Multitasking for iPad
iOS 9 provides support for simultaneous viewing of up to 3 apps at a time on iPads (primary, secondary, and picture-in-picture). iPad Mini 2 and later and iPad Air and later support “Slide Over”, a mode where a second app can be temporarily slid over from the right side of the screen atop a fullscreen app, during which time that primary app is grayed out. The newest and most powerful iPads (iPad Mini 4, iPad Air 2, and iPad Pro) support a second, more powerful form of multitasking called “Split View”, where two apps can be positioned side by side and both are fully active. In iOS 9, camera access is limited to fullscreen apps.
Though multiple apps can run in the foreground simultaneously, the iOS front and rear cameras may only be used by one application at a time. Existing 3rd party apps that use the camera assume they have full access to the camera and sufficient resources to run a smooth camera preview, take beautiful pictures, and capture high-resolution, glitch-free videos. If two apps are permitted to share the camera, the expectation of full access to the camera is broken, as either app could perform a focus, exposure, zoom, or other operation on the camera at an inopportune time for the other app. The presence of a secondary app on screen (even a non camera-using app) may be sufficient to overload system resources needed by the camera to deliver the highest quality recordings and pictures. With these restrictions in mind, we invite camera-using applications to consider which is more important for their user experience: camera usage, or multitasking support? If your app is camera centric, you should add the UIRequiresFullscreen=YES key to your app’s info.plist. If camera usage is a peripheral feature in your app, you’ll likely want to adopt iPad multitasking, but be aware that while your app is running in a multitasking environment, the camera will be unavailable, and you’ll need to detect this and respond appropriately to provide a good user experience. Refer to the updated AVCam sample code for an example of how to react to various AVCaptureSession interruption reasons.


Camera restrictions in iPad Multitasking are slightly different for clients of UIImagePickerController. UIImagePickerController is a stock UIViewController that provides a UI similar to the Apple Camera app. UIImagePickerController clients are allowed full access to the camera if their app is run fullscreen. In the presence of a second (non camera-using) app on screen, they are allowed to run camera preview and take pictures, but not record video. When a user selects Video in the UIImagePickerController view, the camera preview goes black, and the message “(YourApp) must be full screen to record video” is displayed. In the presence of multitasking, if you call -startVideoCapture(), it will return NO.


For clients of AVCaptureSession and AVCaptureDevice, camera usage is strictly limited to one fullscreen app. Be aware that your app may lose access to the camera at any time if a user interacts with his or her iPad to bring up a second multitasking window. When your app loses the camera, your AVCaptureSession receives an AVCaptureSessionWasInterruptedNotification, after which it temporarily stops running. In iOS 9, we’ve enhanced the AVCaptureSessionWasInterruptedNotification. It now provides a userInfo payload dictionary that contains an AVCaptureSessionInterruptionReasonKey. By inspecting the key, you can determine why your camera usage was interrupted, and display appropriate UI feedback if desired. The reasons are:

- Your app is being sent to the background (AVCaptureSessionInterruptionReasonVideoDeviceNotAvailableInBackground).
- Your audio device is now in use by someone else, such as a voice call (AVCaptureSessionInterruptionReasonAudioDeviceInUseByAnotherClient).

- Your camera is in use by someone else (AVCaptureSessionInterruptionReasonVideoDeviceInUseByAnotherClient).
- The user has just invoked multitasking (AVCaptureSessionInterruptionReasonVideoDeviceNotAvailableWithMultipleForegroundApps).


As with pre-existing AVCaptureSession interruptions, the AVCaptureSession remembers that it needs to be restarted when the interruption ends, so even if you take no action, the AVCaptureSession will resume running once the interruption ends. For instance, if a user slides a secondary app over your app while you’re recording a video, your app’s camera preview will stop, as will your recording. When the user dismisses the secondary app view, your AVCaptureSession and preview will restart, though recording does not.



Moving Assets to Photos Library Without Copying

In iOS 8.x and earlier, videos captured in 3rd party applications’ sandboxes must be copied into the Photos Library (Assets Library), in effect halving the available disk storage for capture. New in iOS 9, Photos Library supports moving video assets from your sandbox. We encourage all video capture apps that store media assets to the Photos Library to adopt this new API. Refer to the AVCam sample code (look for [PHPhotoLibrary requestAuthorization: …) for the code to move captured video to the Photos Library.

Replies

Thanks very much for the overview of changes - much appreciated!

One question: Is it correct that for the iPhone 6s rear-facing camera non of the device formats support video HDR anymore?

That's correct. On iPhone 6s and 6s Plus, the rear-facing camera does not support video HDR. The front-facing camera supports video HDR on all formats except for the Photo format (1280x960 or 2576x1932 high-res stills).

how to simulate them? (especially, viewing Live Photo) or Apple had a plan?

How to enable OIS on iPhone 6s plus for 4K video ? I guess 4K port format will not support digital stabilisation so preferredVideoStabilizationMode will be AVCaptureVideoStabilizationModeOff. Please suggest.

As mentioned in the What's New post, there is currently no support for capturing Live Photos, but coming soon in 9.1, there is public API to view and share Live Photos in your apps:

https://developer.apple.com/library/prerelease/ios/releasenotes/General/WhatsNewIniOS/Articles/iOS9_1.html

The 4K video formats support AVCaptureVideoStabilizationModeStandard (not Cinematic). Therefore, if you enable video stabilization by setting your AVCaptureConnection (to MovieFileOutput or VideoDataOutput) preferredVideoStabilizationMode to AVCaptureVideoStabilizationModeStandard or AVCaptureVideoStabilizationModeAuto, the OIS module will be engaged during video capture.

I hear from users that using AVCaptureVideoStabilizationModeStandard results in a crop of video with 4K, is that correct ? If only OIS is used, it shouldn't cause a crop.

Please let me know if using AVCaptureVideoStabilizationModeAuto in 4K preset will enable only OIS without invoking any part of digital stabilisation pipeline ? How does native video app work when it only uses OIS ?

bford, waiting for your response.

OIS during video capture uses a combination of optical and digital video stabilization and DOES lose ~10% field of view. There is no API to only enable the optical side.

I just confirmed from a user and the result is that AVCaptureVideoStabilizationModeStandard does stabilize video in 4K preset but AVCaptureVideoStabilizationModeAuto does not enable any kind of stabilization. Is it a bug ?

Hello bford, I have been scratching my head for years but can not get a workaround for the extreme preview lag. The issue is this. When I have AVCaptureVideoDataOutput set and OpenGLES rendering of frames and use AVCaptureVideoStabilizationModeCinematic, there is considerable delay in the preview. The delay is much much less if I use AVCaptureVideoPreviewLayer. What is the right way here?