>Regarding the vector image, it's been possible for a long time to give Xcode a vector image, but the ability to keep it in vector form in the built bundle is quite new. (Wasn't it introduced at WWDC 2016?) It may well be incompatible with 10.11 or earlier. You might double-check that you've specified the correct target version for the storyboard or XIB in the IB inspector, or you may need to force it to use an older asset catalog format if that's possible. Again, it's not clear where in the documentation to go to find out what macOS runtimes support what features here.
I'm not sure if there is a way to force Xcode to use an older asset catalog format? I'm aware of such a feature for xibs/storyboards but is it possible to do with asset catalogs? The image is not loaded in IB, I use NSImage +imageNamed:. I ended up providing a fallback non-vector image and fallback to it on 10.11. This avoids this crash on 10.11.
>There are a lot of moving parts in what you describe. For example, you're comparing 10.13 and 10.12.6, but are they running on the same physical hardware? (Specifically, if you're running 10.12.6 under virtualization, which is a common way to test older macOS versions, your results may not be comparable.)
In the case of the icns images (generic Mac icon in asset catalog, which creates an icns) it now no longer working as expected when the app is compiled on Xcode 9 (I've had these icons in my asset catalog prior to the release of Xcode 9 and they haven't been changed in awhile and it worked before), it seems the smallest representation is used. I tried to explicitly set the NSImage size to a high value using setSize: and passing in 1024x1024 but the NSImageView appears to still be upscaling a small representation and it looks bad. And in the case of drawing it in a CALayer context, use of CGImageForProposedRect gives me the smallest representation (looks like the 32 point size one).
I haven't been able to run it in the Xcode debugger on 10.12 yet, but my guess is the image loaded from the asset catalog isn't picking up all the representations on 10.12 and earlier, I think they must have changed something that broke the image loading for these resources on 10.12 and earlier. On 10.13 with the exact same image and code, I get the desired behavior I got prior to Xcode 9; the best representation is used when drawn by NSImageView and my CALayer.
To clarify about hardware, I have an iMac still running 10.12 (not testing in a virtual machine) and the small image problem happens. My testing for 10.11 was done on my Macbook Pro in VirtualBox though. Generic icon problem happens in both cases.