How to stop "can't be opened because it is from an unidentified developer"

I've packaged my JavaFX application into a .pkg and signed it using productsign, but when I open it on another computer I get the "This application cannot be opened because it is from an unidentified developer" error. I know it can still be opened by right clicking and pressing open or by changing the security settings, but I don't think this is a good user experience for when I want to sell this app online. How can I stop this from appearing? Why doesn't signing the app stop it?