There should be no expectation of security of binaries. The Notarization process is for distribution of software. Ergo, you wouldn't be notarizing it if you didn't intend to distribute it. If you are distributing it, then anyone to whom you send it could re-distribute it. If they do so illegally, then you are certainly free to take any legal action as necessary.
Of all the people that you might send your files to, Apple is the least likely to allow it to "leak" to "unintended hands". Apple respects the privacy and personal information of all of its users and developers. I would argue that Apple's level of protection of your data is probably is probably higher than what you would find at any other company.
That being said, Apple's first priority is to its customers. If you read your Developer agreement, the section for Notarization says "Apple shall not be responsible for any costs, expenses, damages, losses or other liabilities You may incur as a result of Your Application development, use of the Apple Software, Apple Services (including this digital notary service), or Apple Certificates, tickets, or participation in the Program, including without limitation the fact that Apple performs security checks on Your Application."
What this means is that, although I personally think your binaries are safe with Apple, there are no guarantees.