"codesign wants to access keychain" Always allow does not work

I've recently updated to XCode 11, this now prompts me to enter my password every time I run a build from XCode, I am using a machine as an automated build agent.
Typing my password and pressing "Always allow" does make the prompt go away, but it keeps getting prompted afterwards on each build. This is breaking my automated build process.
It seems like it doesn't remember to "Always Allow" codesign to access keychain.
Any help would be greatly appreciated.