No framework should arbitrarily mark global symbols "public", not matter how blindlingly obvious the definition may seem. This is even more true with operators, where randomly defining new convenience operators is an anti-pattern. (An "offset" function on CGPoint would have been better. Can you go that route in your app? Or use a different operator?)
The short term solution is to remove the "public" from both frameworks, so that the operator functions have "internal" access. (You can't currently use Swift frameworks in binary form, so you should have access to the source code.) You should also reprimand the framework's owner.
In the longer term, if the problem arises in rare cases, I assume there's a way to remove the global symbol, either by using one of the command line tools, or via linker options, or by editing the framework module map.
I know none of this is very satisfactory, but it's the best I can come up with. Maybe take this up on swift-users at swift.org, and see if there's any crowd wisdom existing on this subject?
Sadly, I want to have my cake and eat it too. I want to define extensions and overloads in my OWN libraries, so that the code is in one place, and can be used by my own apps or other libraries that build against lower level libraries.
So I want to give these extensions/overloads public access so *I* can get to them myself when I use them.
Of course, I just don't want anybody than *me* to do this...
But if several of us want to do this, then sadly none of us can do it safely. In short, I want to be able for my lower level library to say "here are a bunch of overloads/extensions" and then my upper level libraries have an easy way of choosing whether or not to make use of them.
[Oh, and you're right about the offset being better. that was a bad example. pretend those aren't points, but vectors, where + would actually make sense.]
If you control the frameworks (or if the frameworks owner is amenable to rational suggestions), I'd say that the convenience functions should be provided in source code form, for clients to use or not use as the case may be. In cases where the implementation is some private algorithm, the operator function could call out to a named function in the frameworks.
I can't imagine why any publicly-offered framework (other than Foundation) would expect to be able to include convenience operators on a type it didn't define, such as CGPoint. It doesn't own the type, and so "owning" the convenience operators is going to lead to tears in exactly the way you described.
Even in the case of vector types (such as the SIMD types), I think it's a mistake for a 3rd-party framework to add public "+" overloads, since it doesn't own the types in that case either.
Again, ask about this on swift-users. I'm sure the clever minds over there have more evolved answers than I do.