4 Replies
      Latest reply on Apr 24, 2019 9:55 AM by mevindhun
      d_ripl Level 1 Level 1 (0 points)

        I am trying to get a custom image loaded as a depth map for an image.

        I thought I could load an regular image from a url as a CIImage and then load the depth image as a separate CIImage. Then all I need to do was use CIContext to write the combined image (based on the WWDC session "Image Editing with Depth").

         

         

            CIImage *theSourceImage = [CIImage imageWithContentsOfURL:theImageUrl];
            CIImage *theDepthImage = [CIImage imageWithContentsOfURL:theDepthUrl  ];
            NSURL *theOutputUrl = [self getOutputURL];
            CIContext *theCIContext = [CIContext context];
            NSError* theWriteError;
            BOOL theResult = [theCIContext writeJPEGRepresentationOfImage:theImageWithoutDepth
                                                                    toURL:theOutputUrl
                                                               colorSpace:CGColorSpaceCreateDeviceRGB()
                                                                  options:@{kCIImageRepresentationDisparityImage: theDepthOnly,
                                                                            }
                                                                    error: &theWriteError];

         

        However when I do this it write the image but also logs a warning in the XCode console:

        addAuxiliaryDataInfo:2720: *** 'auxiliaryDataInfoDictionary' is missing 'kCGImageAuxiliaryDataInfoMetadata' key

         

        I think this indicates that my image that I am trying to use as a disparity image is not formatted correctly but I am uncertain of how to convert it into the proper structure so that I can combine them together.

         

         

        If instead I take an image with depth data in it already I can break it into two pieces and then write a new file to combine them back together and it seems to work fine (no error and the photo is recognized as having depth data in it):

            CIImage *theDepthOnly = [CIImage imageWithContentsOfURL: theImageWithDepthUrl options: @{ kCIImageAuxiliaryDepth: @YES } ];
            CIImage *theImageWithoutDepth = [CIImage imageWithContentsOfURL: theImageWithDepthUrl options: @{ kCIImageAuxiliaryDepth: @NO } ];
        
            NSURL *theOutputUrl = [self getOutputURL];
            CIContext *theCIContext = [CIContext context];
            NSError* theWriteError;
            BOOL theResult = [theCIContext writeJPEGRepresentationOfImage:theImageWithoutDepth
                                                                    toURL:theOutputUrl
                                                               colorSpace:CGColorSpaceCreateDeviceRGB()
                                                                  options:@{kCIImageRepresentationDisparityImage: theDepthOnly,
                                                                            }
                                                                    error: &theWriteError];

        I noticed that the depth image extracted from the image that already has depth has a different color space:

        Printing description of theDepthOnly in debugging console:

          affine [1 0 0 -1 0 576] extent=[0 0 768 576] opaque
            colormatch "Linear Gray"_to_workingspace extent=[0 0 768 576] opaque
              provider 7:initWithCVImage Lh alpha_one extent=[0 0 768 576] opaque

         

        where as the initial custom depth image I was trying to use has:

        Printing description of theDepthImage in debugging console:

          affine [1 0 0 -1 0 722] extent=[0 0 1280 722] opaque
            colormatch "Generic Gray Gamma 2.2 Profile"_to_workingspace extent=[0 0 1280 722] opaque
              IOSurface 0x600000a00b60(2) seed:0 L8 alpha_one extent=[0 0 1280 722] opaque

         

         

        Any guidance or advice would be greatly appreciated.

        • Re: Adding custom depth data to an image
          FrankSchlegel Level 3 Level 3 (120 points)

          Search for the Article "Creating Auxiliary Depth Data Manually" in the AVDepthData documentation (not posting a link here to avoid moderation). I think it describes exactly what you need.

            • Re: Adding custom depth data to an image
              d_ripl Level 1 Level 1 (0 points)

              I think I found that article earlier. It definitely has helpful information. It basically says:

               

              "Load the grayscale image into a CVPixelBuffer. Load its base address, attained via CVPixelBufferLockBaseAddress, as data (CFDataRef) and pass it as the kCGImageAuxiliaryDataInfoData value into a dictionary (CFDictionaryRef)."

               

              I was able to create the CVPixelBuffer with my depth image (I can preview it in the debugger) but I got stuck when trying to load the base address into the CFDictionaryRef:

               

              CVPixelBufferLockBaseAddress(thePixelBuffer, 0);
              CFDataRef theData = CVPixelBufferGetBaseAddress(thePixelBuffer);
              CGFloat theWidth = self.myOutputSize.width;
              CGFloat theHeight = self.myOutputSize.height;
              size_t theBytesPerRow = CVPixelBufferGetBytesPerRow(thePixelBuffer);
              
              NSDictionary *theDepthDictionaryDescription = @{
                (NSString *) kCVPixelBufferCGImageCompatibilityKey: @(YES),
                (NSString *) kCGImagePropertyBytesPerRow: @(theBytesPerRow),
                (NSString *) kCVPixelBufferPixelFormatTypeKey: @(kCVPixelFormatType_DepthFloat32),
                (NSString *) kCGImagePropertyWidth: @(theWidth),
                (NSString *) kCGImagePropertyHeight: @(theHeight)
              };
              
              
              NSDictionary *theDepthDictionary = @{
                (NSString*) kCGImageAuxiliaryDataInfoData: (__bridge NSData*) theData,
                (NSString*) kCGImageAuxiliaryDataInfoDataDescription: theDepthDictionaryDescription,
                (NSString*) kCGImageAuxiliaryDataInfoMetadata: [NSNull null]
              };

               

              It fails to create "theDepthDictionary" at runtime with a "Thread 1: EXC_BAD_ACCESS (code=1, address=0xf0d0d0e18)"

               

              So I tried the alternate approach mentioned in the WWDC session since they seemed to indicate it would do all of those steps for adding the depth data in just a few lines.

               

              I feel like I am missing something obvious but not sure.

                • Re: Adding custom depth data to an image
                  FrankSchlegel Level 3 Level 3 (120 points)

                  I'm not quite sure why the creation of the dictionary fails. I guess it's because CVPixelBufferGetBaseAddress does not return an NSData, but a void*. You have to use [NSData dataWithBytes:length:] to create an NSData object from that.

                   

                  However, I think the main issue is not the attaching of the depth data to the image, but the conversion of the depth image to depth data in the first place. The image that you're loading is a "normal" 8-bit grayscale image (Generic Gray Gamma, L8 [luminance-only, 8 bit]), while the requirement for the depth data is that it's either 16 or 32 bit float with a specific format type and semantic (see the article). You need to find a way to convert your grayscale image into that format. For that you also need to know the maximum depth (in meters) for mapping those values accordingly.

                  I think you can achieve that by creating an empty CVPixelBuffer with the required format and the desired dimensions and then use a framework like Metal or vImage to convert the data from one to the other.

                  • Re: Adding custom depth data to an image
                    mevindhun Level 1 Level 1 (0 points)

                    I am also trying to do attach a grayscale image as a depth map to an image. Have you been able to solve it?