17 Replies
      Latest reply: May 10, 2017 10:40 AM by fergieee RSS
      srpoucse Level 1 Level 1 (0 points)

        Hi,

         

        I'm trying to access a website with NTLM protocol. I'm using a NSURLSession API to access resrouces in this website

        First time I am presented with a challenge and when i supply credentials the callback is sent in two modes.


        i) Same HTTP Connection

        ii) A new HTTP Connection


        The NTLM Authorization header is missing when sent on the same HTTP Connection but exists when sent as a new HTTP Connection.


        This below blog mentions that we need to close the connection when we receive the NTLM challenge and send the new request with creds as a new HTTP Connection.

         

        https://blogs.msdn.microsoft.com/chiranth/2013/09/20/ntlm-want-to-know-how-it-works/


        Could you someone please take a look?


        Thanks,

        Ratna.

        • Re: NTLM Challenge response missing Authorization header
          eskimo Apple Staff Apple Staff (7,530 points)

          The NTLM Authorization header is missing when sent on the same HTTP Connection but exists when sent as a new HTTP Connection.

          Are you looking at this from the perspective of the server?  If so, that seems reasonable.  NTLM authenticates connections, not requests, so when the connection is authenticated there’s no longer any need to include the Authorization header.

          Are you concerned about this on a theoretical level?  Or is it causing actual problems?  If so, what are the symptoms of those problems?

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: NTLM Challenge response missing Authorization header
              srpoucse Level 1 Level 1 (0 points)

              Thanks for the reply.

               

              This happens during the NTLM handshake. I believe when we call the completion handler in the challenge as below


              _completionHandler(xxdisposition, xxxcredential);

               

              This request is sent on the same Connection and  request fails.

               

              However when the completion handler's request is sent on different connection if succeeds. Could you please check on what basis the NSURLSession Challege's completion handler request goes on a new TCP connection?


              PS : Its been a issue with all our NTLM customers.

                • Re: NTLM Challenge response missing Authorization header
                  eskimo Apple Staff Apple Staff (7,530 points)

                  Alas, I’m confused by your description of the issue.  Can you walk me through a timeline of the requests sent and the responses received, including the connection the request was sent on and the Authorization and WWW-Authenticate headers applied in each case.

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: NTLM Challenge response missing Authorization header
                      srpoucse Level 1 Level 1 (0 points)

                      Imagine a website with the following resources. Web Sever :  IIS , Authentication protocol : NTLM

                      a.html

                      a.png

                      a.js

                      .....

                       

                      Try loading abc.com/a.html

                       

                      We get a challenge with 401 response code with beow headers

                        WWW-Authenticate: Negotiate

                        WWW-Authenticate: NTLM

                       

                      NTLM Handshake starts

                      Client sends this

                        Authorization: NTLM aaaa

                       

                      Server responds

                        WWW-Authenticate: NTLMaaaaa

                      Client responds

                        Authorization: NTLMxxxx

                       

                      Server responds with 200

                       

                      What we have noticed that when this response to the challenge is sent on the same TCP connection (lets say C1) in which the challenge comes through, it results in a failure. ( There is no authorization header here)

                      But if the response to challenge is sent on new TCP connection lets say C2 it succeeds and gives the right resource. (There is a authorization header in this case).

                        • Re: NTLM Challenge response missing Authorization header
                          eskimo Apple Staff Apple Staff (7,530 points)

                          Try loading abc.com/a.html

                          OK, this timeline makes things much clearer.  Thanks!

                          Alas, I still need to clarify your conclusion.  You wrote:

                          What we have noticed that when this response to the challenge …

                          The response to which challenge?  You’ve outline multiple request/response pairs, so I’m not sure at which step it switches to a different connection.  Perhaps you could post two timelines, one that shows the working case and one that shows the failing case, with each one showing the request/response pairs and their mapping to TCP connections?

                          Sorry to be so picky here but, well, NTLM is a bit of a nightmare and you really need to nail down the details before coming to any conclusions.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: NTLM Challenge response missing Authorization header
                              srpoucse Level 1 Level 1 (0 points)

                              Timelines for working Scenario

                              TCP 1 - request for a.html

                              TCP 1 - response received from server with WWW-negotiate headers and we do get a call back to NSURLSession didReceiveChallenge API

                                           - Now we create NSURLCredential Object with persistence set to NONE

                                           - Call the completion Handler - (1)

                              TCP 2  - The request triggered from the (1) starts on a new TCP Connection with NTLM xxxx

                              TCP 2  - The server responds back NTLM xxxxxxx

                              TCP 2  - The client responds back NTLM xxxxxxxxxxxx

                              TCP 2 -  Server responds back with 200.

                               

                              Timelines for not working  Scenario

                              TCP 1 - request for a.html

                              TCP 1 - response received from server with WWW-negotiate headers and we do get a call back to NSURLSession didReceiveChallenge API

                                           - Now we create NSURLCredential Object with persistence set to NONE

                                           - Call the completion Handler - (1)

                              TCP 1  - The request triggered from the (1) starts on a same TCP Connection with NTLM xxxx but misses the authorization header

                              TCP 1  - The server responds with a 401

                                • Re: NTLM Challenge response missing Authorization header
                                  eskimo Apple Staff Apple Staff (7,530 points)

                                  OK, that’s super clear.  Thanks.

                                  In my experience your “working Scenario” is how NTLM typically works for Apple clients.  Due to the architectural mismatch between NTLM and HTTP, we end up establishing redundant connections rather than authenticating the connection that’s already in place.

                                  Pasted in below is an explanation of why NTLM is on ongoing source of pain on our platforms.  If you have any control over the server, I strongly encourage you to use something else.

                                  Notwithstanding that, NTLM does normally work, so I’m surprised you’re hitting a problem in a straightforward case like this.  Have you tried setting the persistence parameter to .forSession?

                                  Also, what version of iOS was used for these scenarios?  Have you tested any others?

                                  Share and Enjoy

                                  Quinn “The Eskimo!”
                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                  let myEmail = "eskimo" + "1" + "@apple.com"


                                  NTLM authentication does not follow the architecture for HTTP authentication schemes outlined in RFC 7235.  Specifically, NTLM authenticates connections, not requests.  This causes numerous problems:

                                  • NTLM authentication is fundamentally incompatible with HTTP/2 because HTTP/2 uses one connection for multiple requests.

                                  • NTLM authentication is an ongoing source of problems on Apple platforms because our HTTP stack was designed around the RFC 7235 architecture.  Some of these problems are just bugs that need to be fixed, but others are more fundamental.  For example, in many cases NSURLSession will end up creating extra connections just to deal with NTLM’s unusual requirements.

                                  • NTLM authentication is less efficient than standard HTTP authentication.  Specifically, every new NTLM connection requires 2 extra round trips to the server, whereas with standard HTTP authentication those round trips can often be skipped.

                                  If the server you’re talking to supports an RFC 7235 compliant authentication scheme (typically Basic or Digest), or you have control over the server and can enable such a scheme, I strongly recommend you use a standard scheme rather than continuing with NTLM.

                                    • Re: NTLM Challenge response missing Authorization header
                                      srpoucse Level 1 Level 1 (0 points)

                                      Unfortunately we don't have control over the customer's server.

                                      We did try setting the persistence parameter to Session - In this case we do not get a callback for every resource in the same protection space however we end with up "not working" scenario when the completion handler for the challenge is called on same TCP Connection.

                                      It fails in both iOS 9 & iOS 10.

                                      Could you let me know if there is possibility that the completion handler to the challenge is sent on an existing TCP connection? I'm assuming this could be the reason for the failure. On the contrary if a new TCP connection is created when the completion handler is called it works as expected.

                                        • Re: NTLM Challenge response missing Authorization header
                                          eskimo Apple Staff Apple Staff (7,530 points)

                                          At this point I recommend that you file a bug about this issue; it’s clear that NSURLSession’s NTLM support is not working for your server.  Please post your bug number, just for the record.

                                          As to how you can work around this, I don’t have any great suggestions.  NSURLSession does not have a lot of knobs you can twiddle here (the persistence parameter is the only one I can think of, and you’ve already tried that).

                                          Share and Enjoy

                                          Quinn “The Eskimo!”
                                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                          let myEmail = "eskimo" + "1" + "@apple.com"

                                            • Re: NTLM Challenge response missing Authorization header
                                              srpoucse Level 1 Level 1 (0 points)

                                              Thanks so much again for helping out. Created a radar - rdar://29402344.

                                              Let me know if you need anything else to follow up?

                                                • Re: NTLM Challenge response missing Authorization header
                                                  eskimo Apple Staff Apple Staff (7,530 points)

                                                  Created a radar - rdar://29402344.

                                                  Thanks for that.

                                                  Let me know if you need anything else to follow up?

                                                  Given that this problem is not immediately reproducible (I’ve tried NTLM authentication against a server here at Apple and it works just fine for me), it’d be really helpful if you included:

                                                  Share and Enjoy

                                                  Quinn “The Eskimo!”
                                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                                    • Re: NTLM Challenge response missing Authorization header
                                                      srpoucse Level 1 Level 1 (0 points)

                                                      Given that this problem is not immediately reproducible (I’ve tried NTLM authentication against a server here at Apple and it works just fine for me)

                                                           I have few questions here

                                                                *When you get a challenge, you might have created a credential object. What was persistence mode set here?

                                                                *Was the completion handler called on the same TCP connection?

                                                       

                                                      PS : Our infrastructure includes a proxy before it reaches web server. Please find some observations and research that could be helpful.

                                                       

                                                      As mentioned earlier the failure happens only when the completion handler is called on the same TCP connection. This blog below

                                                      https://blogs.msdn.microsoft.com/chiranth/2013/09/20/ntlm-want-to-know-how-it-works/  mentions that the client should close the TCP connection and open a new one.

                                                       

                                                      2. If the client fails or does not support Kerberos, the Negotiate and NTLM header values initiate an NTCR authentication exchange. The client closes the TCP connection, opens a new one, and sends a request that includes an Authorization: NTLM header. This header also includes encoded text that represents the users UserName, ComputerName, and Domain. This text is used by the Windows Security Support Provider Interface (SSPI) to generate the challenge. If the user account is not a local Windows account on the IIS server, the data is passed on to an appropriate domain controller, which then generates the challenge.

                                                       

                                                      Is there a reason why we send the completion handler on the same TCP connection without closing it?

                                                        • Re: NTLM Challenge response missing Authorization header
                                                          srpoucse Level 1 Level 1 (0 points)

                                                          Adding to it, another link related to NTLM. The below link mentions that server should close the connection but in my experience i have not seen IIS servers sending "Connection: close" header when requested for authorized resource for the first time.

                                                           

                                                          http://davenport.sourceforge.net/ntlm.html

                                                           

                                                          1. The server responds with a 401 status, indicating that the client must authenticate. "NTLM" is presented as a supported authentication mechanism via the "WWW-Authenticate" header. Typically, the server closes the connection at this time:
                                                        • Re: NTLM Challenge response missing Authorization header
                                                          srpoucse Level 1 Level 1 (0 points)

                                                          Hey - Any update?

                                                            • Re: NTLM Challenge response missing Authorization header
                                                              eskimo Apple Staff Apple Staff (7,530 points)

                                                              Any update?

                                                              There’s two parts to this:

                                                              • Bug report — You should be able to see the status of your bug report in the Apple Bug Reporter, and our bugs folks will email you if something significant changes.

                                                              • Workarounds — As I mentioned on 21 Nov, I don’t have any good ideas about how you might work around this issue.

                                                              Share and Enjoy

                                                              Quinn “The Eskimo!”
                                                              Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                                              let myEmail = "eskimo" + "1" + "@apple.com"

                                                  • Re: NTLM Challenge response missing Authorization header
                                                    fergieee Level 1 Level 1 (0 points)

                                                    I believe I am running into the same issues. Intermittent NTLM authentication failures. I'd like to verify if it is because of a connection re-use like you called out. How were you able to determine that the connection is re-used?