NTLM Challenge response missing Authorization header

Hi,


I'm trying to access a website with NTLM protocol. I'm using a NSURLSession API to access resrouces in this website

First time I am presented with a challenge and when i supply credentials the callback is sent in two modes.

i) Same HTTP Connection

ii) A new HTTP Connection

The NTLM Authorization header is missing when sent on the same HTTP Connection but exists when sent as a new HTTP Connection.

This below blog mentions that we need to close the connection when we receive the NTLM challenge and send the new request with creds as a new HTTP Connection.


https://blogs.msdn.microsoft.com/chiranth/2013/09/20/ntlm-want-to-know-how-it-works/

Could you someone please take a look?

Thanks,

Ratna.

Replies

The NTLM Authorization header is missing when sent on the same HTTP Connection but exists when sent as a new HTTP Connection.

Are you looking at this from the perspective of the server? If so, that seems reasonable. NTLM authenticates connections, not requests, so when the connection is authenticated there’s no longer any need to include the

Authorization
header.

Are you concerned about this on a theoretical level? Or is it causing actual problems? If so, what are the symptoms of those problems?

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks for the reply.


This happens during the NTLM handshake. I believe when we call the completion handler in the challenge as below

_completionHandler(xxdisposition, xxxcredential);


This request is sent on the same Connection and request fails.


However when the completion handler's request is sent on different connection if succeeds. Could you please check on what basis the NSURLSession Challege's completion handler request goes on a new TCP connection?

PS : Its been a issue with all our NTLM customers.

Alas, I’m confused by your description of the issue. Can you walk me through a timeline of the requests sent and the responses received, including the connection the request was sent on and the

Authorization
and
WWW-Authenticate
headers applied in each case.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Imagine a website with the following resources. Web Sever : IIS , Authentication protocol : NTLM

a.html

a.png

a.js

.....


Try loading abc.com/a.html


We get a challenge with 401 response code with beow headers

WWW-Authenticate: Negotiate

WWW-Authenticate: NTLM


NTLM Handshake starts

Client sends this

Authorization: NTLM aaaa


Server responds

WWW-Authenticate: NTLMaaaaa

Client responds

Authorization: NTLMxxxx


Server responds with 200


What we have noticed that when this response to the challenge is sent on the same TCP connection (lets say C1) in which the challenge comes through, it results in a failure. ( There is no authorization header here)

But if the response to challenge is sent on new TCP connection lets say C2 it succeeds and gives the right resource. (There is a authorization header in this case).

Try loading abc.com/a.html

OK, this timeline makes things much clearer. Thanks!

Alas, I still need to clarify your conclusion. You wrote:

What we have noticed that when this response to the challenge …

The response to which challenge? You’ve outline multiple request/response pairs, so I’m not sure at which step it switches to a different connection. Perhaps you could post two timelines, one that shows the working case and one that shows the failing case, with each one showing the request/response pairs and their mapping to TCP connections?

Sorry to be so picky here but, well, NTLM is a bit of a nightmare and you really need to nail down the details before coming to any conclusions.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Timelines for working Scenario

TCP 1 - request for a.html

TCP 1 - response received from server with WWW-negotiate headers and we do get a call back to NSURLSession didReceiveChallenge API

- Now we create NSURLCredential Object with persistence set to NONE

- Call the completion Handler - (1)

TCP 2 - The request triggered from the (1) starts on a new TCP Connection with NTLM xxxx

TCP 2 - The server responds back NTLM xxxxxxx

TCP 2 - The client responds back NTLM xxxxxxxxxxxx

TCP 2 - Server responds back with 200.


Timelines for not working Scenario

TCP 1 - request for a.html

TCP 1 - response received from server with WWW-negotiate headers and we do get a call back to NSURLSession didReceiveChallenge API

- Now we create NSURLCredential Object with persistence set to NONE

- Call the completion Handler - (1)

TCP 1 - The request triggered from the (1) starts on a same TCP Connection with NTLM xxxx but misses the authorization header

TCP 1 - The server responds with a 401

OK, that’s super clear. Thanks.

In my experience your “working Scenario” is how NTLM typically works for Apple clients. Due to the architectural mismatch between NTLM and HTTP, we end up establishing redundant connections rather than authenticating the connection that’s already in place.

Pasted in below is an explanation of why NTLM is on ongoing source of pain on our platforms. If you have any control over the server, I strongly encourage you to use something else.

Notwithstanding that, NTLM does normally work, so I’m surprised you’re hitting a problem in a straightforward case like this. Have you tried setting the

persistence
parameter to
.forSession
?

Also, what version of iOS was used for these scenarios? Have you tested any others?

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

NTLM authentication does not follow the architecture for HTTP authentication schemes outlined in RFC 7235. Specifically, NTLM authenticates connections, not requests. This causes numerous problems:

  • NTLM authentication is fundamentally incompatible with HTTP/2 because HTTP/2 uses one connection for multiple requests.

  • NTLM authentication is an ongoing source of problems on Apple platforms because our HTTP stack was designed around the RFC 7235 architecture. Some of these problems are just bugs that need to be fixed, but others are more fundamental. For example, in many cases NSURLSession will end up creating extra connections just to deal with NTLM’s unusual requirements.

  • NTLM authentication is less efficient than standard HTTP authentication. Specifically, every new NTLM connection requires 2 extra round trips to the server, whereas with standard HTTP authentication those round trips can often be skipped.

If the server you’re talking to supports an RFC 7235 compliant authentication scheme (typically Basic or Digest), or you have control over the server and can enable such a scheme, I strongly recommend you use a standard scheme rather than continuing with NTLM.

Unfortunately we don't have control over the customer's server.

We did try setting the persistence parameter to Session - In this case we do not get a callback for every resource in the same protection space however we end with up "not working" scenario when the completion handler for the challenge is called on same TCP Connection.

It fails in both iOS 9 & iOS 10.

Could you let me know if there is possibility that the completion handler to the challenge is sent on an existing TCP connection? I'm assuming this could be the reason for the failure. On the contrary if a new TCP connection is created when the completion handler is called it works as expected.

At this point I recommend that you file a bug about this issue; it’s clear that NSURLSession’s NTLM support is not working for your server. Please post your bug number, just for the record.

As to how you can work around this, I don’t have any great suggestions. NSURLSession does not have a lot of knobs you can twiddle here (the

persistence
parameter is the only one I can think of, and you’ve already tried that).

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thanks so much again for helping out. Created a radar - rdar://29402344.

Let me know if you need anything else to follow up?

Created a radar - rdar://29402344.

Thanks for that.

Let me know if you need anything else to follow up?

Given that this problem is not immediately reproducible (I’ve tried NTLM authentication against a server here at Apple and it works just fine for me), it’d be really helpful if you included:

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Given that this problem is not immediately reproducible (I’ve tried NTLM authentication against a server here at Apple and it works just fine for me)

I have few questions here

*When you get a challenge, you might have created a credential object. What was persistence mode set here?

*Was the completion handler called on the same TCP connection?


PS : Our infrastructure includes a proxy before it reaches web server. Please find some observations and research that could be helpful.


As mentioned earlier the failure happens only when the completion handler is called on the same TCP connection. This blog below

https://blogs.msdn.microsoft.com/chiranth/2013/09/20/ntlm-want-to-know-how-it-works/ mentions that the client should close the TCP connection and open a new one.


2. If the client fails or does not support Kerberos, the Negotiate and NTLM header values initiate an NTCR authentication exchange. The client closes the TCP connection, opens a new one, and sends a request that includes an Authorization: NTLM header. This header also includes encoded text that represents the users UserName, ComputerName, and Domain. This text is used by the Windows Security Support Provider Interface (SSPI) to generate the challenge. If the user account is not a local Windows account on the IIS server, the data is passed on to an appropriate domain controller, which then generates the challenge.


Is there a reason why we send the completion handler on the same TCP connection without closing it?

Adding to it, another link related to NTLM. The below link mentions that server should close the connection but in my experience i have not seen IIS servers sending "Connection: close" header when requested for authorized resource for the first time.😟


http://davenport.sourceforge.net/ntlm.html


The server responds with a 401 status, indicating that the client must authenticate. "NTLM" is presented as a supported authentication mechanism via the "WWW-Authenticate" header. Typically, the server closes the connection at this time:

Hey - Any update?

Any update?

There’s two parts to this:

  • Bug report — You should be able to see the status of your bug report in the Apple Bug Reporter, and our bugs folks will email you if something significant changes.

  • Workarounds — As I mentioned on 21 Nov, I don’t have any good ideas about how you might work around this issue.

Share and Enjoy

Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"