SSL handshake failure when connecting to Apple Pay gateway

I'm trying to connect to the Apple gateway to get through the merchant validation process but it is not working on .NET. For starters, when I call the payment gateway using CURL it works fine, I used the following command:

curl --data '{"merchantIdentifier":"merchant.########", "domainName":"########.net", "displayName":"merchant.########"}' -H "Content-Type: application/json" -X POST --cert ./apple-pay-cert.pem https://apple-pay-gateway.apple.com/paymentservices/paymentSession

and it successfully returns the merchant session back. However, I'm trying to do it on .NET and I've simplified the code here:

var x509 = new X509Certificate2(System.IO.File.ReadAllBytes("apple-pay-cert.pem"));
            ServicePointManager.Expect100Continue = true;
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
            X509Certificate2 certificate = new X509Certificate2("apple-pay-cert.pem", "",
            X509KeyStorageFlags.UserKeySet | X509KeyStorageFlags.PersistKeySet |
            X509KeyStorageFlags.Exportable);
            var handler = new HttpClientHandler();
            //var certificate = new X509Certificate2(fileName: "apple-pay-cert.pem", password: "");

            handler.ClientCertificates.Add(x509);

            using (var httpClient = new HttpClient())
            {
                using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://apple-pay-gateway.apple.com/paymentservices/paymentSession"))
                {
                    var response = await httpClient.SendAsync(request);
                }
            }

I removed the body/data part to keep it simple. I have tried every variation of this code, reading the certificate from file then passing it to handler, or directly assigning the certificate, or changing the security protocol to accept only TLS 1.2 channels.

The error I get is almost always the same:

An unhandled exception has occurred while executing the request.
      System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
       ---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
       ---> Interop+AppleCrypto+SslException: handshake failure
         --- End of inner exception stack trace ---
         at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         --- End of inner exception stack trace ---
         at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
         at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)

EDIT: I was able to make it work. Kinda. I used a different http handler called SocketsHttpHandler like this:

var cert = new X509Certificate2("apple-pay-cert.pem", "",
            X509KeyStorageFlags.UserKeySet | X509KeyStorageFlags.PersistKeySet |
            X509KeyStorageFlags.Exportable);
            var sslOptions = new SslClientAuthenticationOptions();
            var shHandler = new SocketsHttpHandler
            {
                MaxConnectionsPerServer = 100,
                AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate,
                PooledConnectionLifetime = TimeSpan.FromMinutes(1),
                ConnectTimeout = TimeSpan.FromSeconds(10),
                PooledConnectionIdleTimeout = TimeSpan.FromSeconds(10),
                ResponseDrainTimeout = TimeSpan.FromSeconds(10),
            };

            if (cert != null)
            {
                shHandler.SslOptions = new SslClientAuthenticationOptions()
                {
                    ClientCertificates = new X509CertificateCollection(),
                };

                shHandler.SslOptions.ClientCertificates.Add(cert);
                shHandler.SslOptions.LocalCertificateSelectionCallback = (object sender, string targetHost, X509CertificateCollection localCertificates, X509Certificate remoteCertificate, string[] acceptableIssuers) => cert;
            }

            using (var httpClient = new HttpClient(shHandler))
            {
                using (var request = new HttpRequestMessage(new HttpMethod("POST"), "https://apple-pay-gateway.apple.com/paymentservices/paymentSession"))
                {
                   
                    var response = await httpClient.SendAsync(request);
                    Console.WriteLine(response.Content.ReadAsStringAsync().Result);
                }
            }

Add in the body of the request and I was able to return a usable merchant session. The only problem is that it prompts me to enter my mac's login credentials every time I run the code to access the certificate from the keychain, how can I prevent that?