AppProxyProvider is not happy with some website

Question

Created Apr ’21

Replies 5

Boosts 0

Participants 2

Hi there,

I am using AppProxyProvider. It seems that there is some problem to visit some website going through appproxy even if it is only pass through all traffics.

How to reproduce:
With AppProxyProvider, setup the filter rule and capture only tcp 80 and 443. Then passthrough all traffic between tcp flow and remote connection. Then try to visit site:

Code Block http://13.210.37.51

With chrome browser, there is no content shown.
But if visit the same site without going through appproxy, all good.

Digging more into the trace got, it seems might have something to do with below error:

Code Block ⛔️FZ$--read from TCP connection error: Optional(Error Domain=kNWErrorDomainPOSIX Code=96 "No message available on STREAM" UserInfo={NSDescription=No message available on STREAM}) Optional(13.210.37.51:80)	fzmacappproxy	11:18:16.988853+1000

Thanks in advance for any suggestion.

Boost

Answer 1

Systems Engineer OP

Apple

Apr ’21

Somewhere in your process you are hitting ENODATA at the TCP level. A few things I would double check:

1) What are you getting for the remoteEndpoint for the NEAppProxyTCPFlow in handleNewFlow?

2) After you open the remote side of the connection and open the local flow, where is this happening in your flow copying process?

3) Does the same thing happen when you use a hostname?

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

0

Answer 2

richard_wang OP

May ’21

1) What are you getting for the remoteEndpoint for the NEAppProxyTCPFlow in handleNewFlow?

Code Block 🤪FZ$--hostName: dns-block-page-sm.familyzone.com remoteEp: 13.210.37.51:80 appId: com.google.Chrome.helper	fzmacappproxy	09:43:24.523718+1000

2) After you open the remote side of the connection and open the local flow, where is this happening in your flow copying process?

We use NWTCPConnection instead, but the logic is the same.

Code Block 	private func readFromTcpConnection(tcpflow: NEAppProxyTCPFlow) -> Void {
        ...
		if !self.waitForTCPConnectionToBeReady(connection) {
			flowClose(tcpFlow: tcpflow, error: nil, connStruct: connStruct)
			return
		}
		connection.readMinimumLength(1, maximumLength: 65535, completionHandler: { [unowned self] (data, error) in
			guard error == nil else {
				FZLog(.error, "read from TCP connection error: \(error.debugDescription) \(connection.remoteAddress.debugDescription)")
				tcpflow.closeWriteWithError(error)
				tcpflow.closeReadWithError(error)
				connection.cancel()
				return
			}
			guard let readData = data else {
				FZLog(.error, "read from TCP connection error: \(connection.remoteAddress.debugDescription)")
				tcpflow.closeWriteWithError(error)
				tcpflow.closeReadWithError(error)
				connection.cancel()
			}
			tcpflow.write(readData) { [unowned self] error in
				guard error == nil else {
					FZLog(.error, "write to TCP flow error: \(error.debugDescription)")
					tcpflow.closeWriteWithError(error)
					tcpflow.closeReadWithError(error)
					connection.cancel()
					return
				}
				FZLog(.verbose, "write \(readData.count) bytes to TCP flow from \(connection.remoteAddress.debugDescription)")
				self.readFromTcpConnection(tcpflow: tcpflow)
			}
		})
	}

3) Does the same thing happen when you use a hostname?
No difference when I use the hostname:

Code Block http://dns-block-page-sm.familyzone.com/

0

Answer 3

Systems Engineer OP

Apple

May ’21

If you try swapping out NWTCPConnection for NWConnection here are you able to see isComplete being used in the receive/read logic closure? For example, if isComplete is set to true do not attempt another read and close the flow and cancel the connection, otherwise try another read.

Code Block swiftconnection.receive(minimumIncompleteLength: 1, maximumLength: 2048) { (data, _, isComplete, error) in
	// Incorporate isComplete here in your 
}

If that does not work you may want to take a look at a packet trace also.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

0

Answer 4

richard_wang OP

May ’21

Some progress:
After updated the code like below, that web page show up.

Only thing remains is: the browser keeps loading this web page: waiting for ....
Seen from the traces, new flowId keeps generated.

Code Block 		connection.readMinimumLength(1, maximumLength: 65535, completionHandler: { [unowned self] (data, error) in
			guard error == nil else {
				FZLog(.error, "read from TCP connection error: \(error.debugDescription) \(connection.remoteAddress.debugDescription)")
				// Close down only when read complete
				if let error = error {
					if error._code == ECANCELED {
						FZLog(.verbose, "TCP connection cancelled: \(connection.debugDescription)")
						tcpflow.closeWriteWithError(error)
						tcpflow.closeReadWithError(error)
						connection.cancel()
					}
				}
				return
			}
...

0

Answer 5

Systems Engineer OP

Apple

May ’21

That does look like progress. Do you know in this case what connections are failing? For example, there was some issues with wss:// connections and some types of Ajax cross origin client side requests that were addressed in NETransparentProxyProvider on Big Sur.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

0