As an alternative you could use the CoreImage kMeans filter - the code is quite a bit simpler.
https://developer.apple.com/documentation/coreimage/cifilter/3547110-kmeansfilter
The example code there shows how to run it an generate an image that has been mapped to the colors.
Once you've done that then you can run through each pixel in the mapped image and compare it with the palette colors.