Architecture-specific defines and/or libraries using CMake

Most parts of our application are built using CMake because they are cross-platform. Now, I know I can set CMAKE_OSX_ARCHITECTURES to build a a fat binary, and it works fine for our smaller, self-contained components. However, some components use optimized third-party libraries that are architecture specific. There's really two cases here:

1. The library interface is standard (e.g., BLAS) and we're just using a different library to provide the implementation.

2. Our own code path depends on the library, in which case we pass in a define (using target_compile_definitions) know which code to use.

Is there a way to make the linker libraries or compile definitions depend on the target architecture? For the first case, I might try listing both and hoping only the needed slice from each one is used (though that presuppose either that they each only have one slice, or that the single-slice one comes first). But in the second case, I really don't see how to do this, short of having two separate thin targets that are manually lipo'd together.