Adding custom depth data to an image

I am trying to get a custom image loaded as a depth map for an image.

I thought I could load an regular image from a url as a CIImage and then load the depth image as a separate CIImage. Then all I need to do was use CIContext to write the combined image (based on the WWDC session "Image Editing with Depth").



    CIImage *theSourceImage = [CIImage imageWithContentsOfURL:theImageUrl];
    CIImage *theDepthImage = [CIImage imageWithContentsOfURL:theDepthUrl  ];
    NSURL *theOutputUrl = [self getOutputURL];
    CIContext *theCIContext = [CIContext context];
    NSError* theWriteError;
    BOOL theResult = [theCIContext writeJPEGRepresentationOfImage:theImageWithoutDepth
                                                            toURL:theOutputUrl
                                                       colorSpace:CGColorSpaceCreateDeviceRGB()
                                                          options:@{kCIImageRepresentationDisparityImage: theDepthOnly,
                                                                    }
                                                            error: &theWriteError];


However when I do this it write the image but also logs a warning in the XCode console:

addAuxiliaryDataInfo:2720: *** 'auxiliaryDataInfoDictionary' is missing 'kCGImageAuxiliaryDataInfoMetadata' key


I think this indicates that my image that I am trying to use as a disparity image is not formatted correctly but I am uncertain of how to convert it into the proper structure so that I can combine them together.



If instead I take an image with depth data in it already I can break it into two pieces and then write a new file to combine them back together and it seems to work fine (no error and the photo is recognized as having depth data in it):

    CIImage *theDepthOnly = [CIImage imageWithContentsOfURL: theImageWithDepthUrl options: @{ kCIImageAuxiliaryDepth: @YES } ];
    CIImage *theImageWithoutDepth = [CIImage imageWithContentsOfURL: theImageWithDepthUrl options: @{ kCIImageAuxiliaryDepth: @NO } ];

    NSURL *theOutputUrl = [self getOutputURL];
    CIContext *theCIContext = [CIContext context];
    NSError* theWriteError;
    BOOL theResult = [theCIContext writeJPEGRepresentationOfImage:theImageWithoutDepth
                                                            toURL:theOutputUrl
                                                       colorSpace:CGColorSpaceCreateDeviceRGB()
                                                          options:@{kCIImageRepresentationDisparityImage: theDepthOnly,
                                                                    }
                                                            error: &theWriteError];

I noticed that the depth image extracted from the image that already has depth has a different color space:

Printing description of theDepthOnly in debugging console:

  affine [1 0 0 -1 0 576] extent=[0 0 768 576] opaque
    colormatch "Linear Gray"_to_workingspace extent=[0 0 768 576] opaque
      provider 7:initWithCVImage Lh alpha_one extent=[0 0 768 576] opaque


where as the initial custom depth image I was trying to use has:

Printing description of theDepthImage in debugging console:

  affine [1 0 0 -1 0 722] extent=[0 0 1280 722] opaque
    colormatch "Generic Gray Gamma 2.2 Profile"_to_workingspace extent=[0 0 1280 722] opaque
      IOSurface 0x600000a00b60(2) seed:0 L8 alpha_one extent=[0 0 1280 722] opaque



Any guidance or advice would be greatly appreciated.

Up vote post of d_ripl Down vote post of d_ripl
2.4k views
Posted by
d_ripl
Reply
Add a Comment

Replies

Search for the Article "Creating Auxiliary Depth Data Manually" in the 

AVDepthData
documentation (not posting a link here to avoid moderation). I think it describes exactly what you need.
Posted by
FrankSchlegel
Up vote reply of FrankSchlegel Down vote reply of FrankSchlegel
Add a Comment

I think I found that article earlier. It definitely has helpful information. It basically says:


"Load the grayscale image into a CVPixelBuffer. Load its base address, attained via CVPixelBufferLockBaseAddress, as data (CFDataRef) and pass it as the kCGImageAuxiliaryDataInfoData value into a dictionary (CFDictionaryRef)."


I was able to create the CVPixelBuffer with my depth image (I can preview it in the debugger) but I got stuck when trying to load the base address into the CFDictionaryRef:


CVPixelBufferLockBaseAddress(thePixelBuffer, 0);
CFDataRef theData = CVPixelBufferGetBaseAddress(thePixelBuffer);
CGFloat theWidth = self.myOutputSize.width;
CGFloat theHeight = self.myOutputSize.height;
size_t theBytesPerRow = CVPixelBufferGetBytesPerRow(thePixelBuffer);

NSDictionary *theDepthDictionaryDescription = @{
  (NSString *) kCVPixelBufferCGImageCompatibilityKey: @(YES),
  (NSString *) kCGImagePropertyBytesPerRow: @(theBytesPerRow),
  (NSString *) kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_DepthFloat32),
  (NSString *) kCGImagePropertyWidth: @(theWidth),
  (NSString *) kCGImagePropertyHeight: @(theHeight)
};


NSDictionary *theDepthDictionary = @{
  (NSString*) kCGImageAuxiliaryDataInfoData: (__bridge NSData*) theData,
  (NSString*) kCGImageAuxiliaryDataInfoDataDescription: theDepthDictionaryDescription,
  (NSString*) kCGImageAuxiliaryDataInfoMetadata: [NSNull null]
};


It fails to create "theDepthDictionary" at runtime with a "Thread 1: EXC_BAD_ACCESS (code=1, address=0xf0d0d0e18)"


So I tried the alternate approach mentioned in the WWDC session since they seemed to indicate it would do all of those steps for adding the depth data in just a few lines.


I feel like I am missing something obvious but not sure.

Posted by
d_ripl
Up vote reply of d_ripl Down vote reply of d_ripl
Add a Comment

I'm not quite sure why the creation of the dictionary fails. I guess it's because 

CVPixelBufferGetBaseAddress
does not return an 
NSData
, but a
void*
. You have to use 
[NSData dataWithBytes:length:]
to create an 
NSData
object from that.


However, I think the main issue is not the attaching of the depth data to the image, but the conversion of the depth image to depth data in the first place. The image that you're loading is a "normal" 8-bit grayscale image (Generic Gray Gamma, L8 [luminance-only, 8 bit]), while the requirement for the depth data is that it's either 16 or 32 bit float with a specific format type and semantic (see the article). You need to find a way to convert your grayscale image into that format. For that you also need to know the maximum depth (in meters) for mapping those values accordingly.

I think you can achieve that by creating an empty 

CVPixelBuffer
with the required format and the desired dimensions and then use a framework like Metal or vImage to convert the data from one to the other.
Posted by
FrankSchlegel
Up vote reply of FrankSchlegel Down vote reply of FrankSchlegel
Add a Comment

I am also trying to do attach a grayscale image as a depth map to an image. Have you been able to solve it?

Posted by
mevindhun
Up vote reply of mevindhun Down vote reply of mevindhun
Add a Comment