Using an image to pass a location map from one filter t'other

Hi rowlands,

First let me tell you that I can understand your frustration with converting your existing filters to Metal. The documentation is realy spotty at best and in most cases it just helps to read function signatures and guess...

Concerning your question: Check out the

init(bitmapData: Data, bytesPerRow: Int, size: CGSize, format: CIFormat, colorSpace: CGColorSpace?)

The color space that the image is defined in. It must be a Quartz 2D color space (

CGColorSpace

nil

You should use that instead of creating a

CGImage

If that alone doesn't help, you can also try to pass your location data in a

CISampler

CIImage

CISampler

nil

Hope that helps!

Frank,

Thanks man... I've always done it through CGImage first and so I never thought to look at creating a CIImage directly. I did attempt to create a CGImage with no color space, but I got no image instead 🙂

I never knew that you could create a sampler directly either!

I really appreciate your help Frank, you've given me hope with these two suggestions. You've helped save my concrete wall from getting new dents.

Glad I could help!

Let me know how it went. I'm also in the middle of porting lots of filters to CI + Metal, so every lesson learned and shared is helpful.

Edit 3: I've got it working, 4 days I've spent on this and finally I have it working.

You need to add 0.5 to the vector when reading from the map.

location = sample( map, vec2( float( c ) + 0.5, 0.5 ) );

My first try was to create a CIIImage without using a CGImage, and including no color space. I get the same issue.

I missunderstood about the CISampler, I thought I could create a sampler directly from a block of memory... I did try creating a sampler and passing in some options ( kCISamplerWrapMode = kCISamplerWrapClamp, kCISamplerFilterMode = kCISamplerFilterNearest ). Didn't help.

I even swapped the image dimensions, so instead of being a tall ( 1 x 6 ), it's now a wide ( 6 x 1 ) image. No difference.

Here's what I have noticed however, I set all the x co-ordinates to be 127 (I'm using a 8-Bit image so I can simply multiple the values by 255.0 to get the locations. They all appear to be in the horizontal center, even if the vertical points are still off by 23 (except 0).

I've even resorted to disabling color management on the context and still getting the same issue. It's like there some missing magic, that's causing the values to get adjusted or...

Maybe there's something wrong with my kernel; it's pretty simple... By creating a CIImage directly from a block of memory, the channel order is different, so it's now reading the horizontal location from the alpha channel.

kernel vec4 pdsMap(sampler image, __table sampler map, float sampleCount)
{
    vec2 d = destCoord();
    
    vec3 r = sample(image, samplerCoord(image)).rgb;
    vec4 location;
    
    int count = int( sampleCount );
    
    for ( int c = 0; c < count; c++ ) {
        location = sample( map, vec2( c, 0 ) );
        location.a = location.a * 255.0;
        location.r = location.r * 255.0;

        if ( abs( d.x - location.a ) + abs( d.y - location.r ) < 1.0 ) { r = vec3( 1.0, 1.0, 1.0 ); }
    }
    
    return vec4( r, 1.0 );
}

edit: Thanks Frank for taking the time to help me out; I know that you're busy also.

edit 2: In the last testing I've just done, the vertical locations are now off by 26 pixels, except 0! It's really odd. Oh just for comparison sake, I hardcoded the locations as an array in the kernel and used the same 'testing function" and it showed them being in the correct locations.

Did you already try to create a

CIImage

init(bitmapData: Data, bytesPerRow: Int, size: CGSize, format: CIFormat, colorSpace: CGColorSpace?)

CGImage

let map = CIImage(bitmapData: data, 
                  bytesPerRow: data.length,
                  size: CGSize(width: <numPoints>, height: 1),
                  format: .RG8, // two 8-bit components
                  colorSpace: nil)

Also two observations from your kernel:

• You don't need to pass

  sampleCount

  . You can just ask for

  map.size.y

  .
• Sample

  image

  only in the else case, that prevents unneeded texture fetches.

What does

__table

I also just realized that this kernel is still written in CIKL. Didn't you want to switch to Metal?

Yes, I did try it by using the function [CIImage initWithBitmapData:] and while it didn't solve the problem, it does remove the step of using a CGImage, so I really apprecaite it.

I was passing sampleCount directly to the kernel for two reasons;

1, in my experiements I was trying different image dimensions (I know vImage works best with image sizes that are multiplies of 4).

2, I intend to store multiple maps in a single image, and I felt that it I pass that information to the kernel, it reduces the kernel having to read that data for every pixel.

What does

__table

"Using the

__table

envmap

Other documentation states:

"On macOS 10.10 and earlier, you could qualify sampler parameters as __table to alter the behavior of ROI callbacks for that sampler, but now this qualifier has no effect."

In my desperation, I tried it, it didn't help, but I didn't remove it either.

I also just realized that this kernel is still written in CIKL. Didn't you want to switch to Metal?

Not really; if it ain't broke an all. However I'm trying to design my filter so that it *should* be easier to convert to Metal when I need to (which means I can't dynamically create a kernel with the locations in it). I am also hoping that by doing it this way, I'll improve performance, reduce memory overhead, but I do know it will come at the cost of additional texture reads.

Thanks for the pointer about RB8, I just checked the headers and lo... Since 10.11, awesome. There's an RG16, which is probably what I'll end up using otherwise my locations will be limited to 255 x 255 🙂

Thanks Frank, I really appreciate your help with this. Hopefully it might help others.

Thanks for the tip about kCIFormatRG16.

I now have it working with 16-Bit locations, the byte order of the data must be Intel/LittleEndian (where as I recall float must be Motorola/BigEndian).

I also seperated the kernel into two, this way (in theory) it only needs to do the expansion once and not for each and every pixel when utilizing the locations on the actual image.

kernel vec4 pdsMapExpander( sampler image )
{
    vec4 location = sample( image, samplerCoord( image ) );
    location.r *= 65535.0;
    location.g *= 65535.0;

    return location;
}

And the "plotter" kernel.

kernel vec4 pdsMapPlotter(sampler image, __table sampler map, float sampleCount)
{
    vec2 d = samplerCoord(image);
    
    vec3 r = sample( image, d ).rgb;
    vec4 location;
    
    int count = int( sampleCount );
    
    for ( int c = 0; c < count; c++ ) {
        location = sample( map, vec2( float( c ) + 0.5, 0.5 ) );

        if ( abs( d.x - location.r ) + abs( d.y - location.g ) < 1.0 ) { r = vec3( 1.0, 1.0, 1.0 ); }
    }
    
    return vec4( r, 1.0 );
}

Hopefully this will help others.