I'm using a Mac Mini as a Jenkins agent, which we use to run our Xcode tests on physical iOS devices. It's configured for remote access with SSH & screen sharing with VNC. Every few days they start failing completely. Sometimes one of them is up a bit longer, but more usually they're both down.
If I look in the GUI it says it's running. The correct ports are listening.
In Console I can see that the sshd process exits with 255 the instant it's started, but I haven't been able to get anything more specific.
I've found that I can get SSH & VNC access back with
launchctl bootout system/<svc name>
launchctl disable system/<svc name>
launchctl enable system/<svc name>
launchctl bootstrap system <plist file name>
The problem is that I can't tell from the remote device that it's not accessible by SSH/VNC. The different interfaces say that sure, everything's fine.
When I do a launchctl print there are some differences between the non-working and working versions. I don't know if these are actual indicators that it's down, or artifacts of the way I restarted them. The differences are consistent for both VNC & SSH:
Not working but apparently running:
path = (submitted by smd.215)
submitted job. ignore execute allowed.
system service = 0
Working after launchctl stop/restart:
path = /System/Library/LaunchDaemons/<plist file>
system service = 1
So, a few questions:
Has anyone else seen this?
Is there some way to get more error information about why sshd is exiting in the logs/console?
Is there a way to detect that sshd is failing, even though there's no system log entry for the failure, and the various interfaces show that everything's fine?
What's the cleanest way to tell the system to restart remote access every day just in case it can't be identified any other way?
Post
Replies
Boosts
Views
Activity
One of our customers is having a problem finding our macOS app in the app store on an M1 system. It just doesn't show up, and if they use a direct link that I've sent them it shows a "not compatible with your system" message.
The app is currently compiled for Intel. We were missing some compatible CocoaPods & libraries for a universal build until just recently.
We know that it runs fine under Rosetta 2.
When they look at the list of compatible systems it says:
Requires
a Mac running macOS 10.15 or
later with an Intel 64-bit processor or
additional software not available in your
country or region.
They're on macOS Monterey, so I can't think of what the "additional software" could be except for Rosetta 2.
They also say that they already have Rosetta 2 installed, and I haven't seen the App Store block a download because of missing Rosetta 2.
How can I check to see that they do, in fact, have Rosetta 2 installed?
Has anyone seen the App Store block a download because of Rosetta 2?
Any ideas what could be causing this other than Rosetta 2?
The customer's region is New York, USA, so "not available in your
country or region." is unlikely.
When I do a search from my M1 mini the app shows up just fine.
I'm seeing the connection to the VPN gateway failing in our Network Extension (not a System Extension) most of the time. Sometimes it succeeds. There's no difference in what the application or the extension are doing in the two cases.
I can't see a pattern to when it fails, but In the console I see different messages. The only thing I've seen showing up consistently on failures but not successes is the message about the swfs_pid_entry.
On failure:
vpn_extension Gateway address 10.10.10.10, port 443
kernel ALF, old data swfs_pid_entry <private>, updaterules_msg <private>, updaterules_state <private>
vpn_extension connect failed with error 65 (No route to host)
kernel connect() - failed necp_set_socket_domain_attributes
vpn_extension Connect returncode 65
On success:
vpn_extension Gateway address 10.10.10.10, port 443
trustd User has disabled system data installation.
We're seeing an accessibility error at several points in our tests
Failed to get list of active applications: Accessibility error kAXErrorServerNotFound from AXUIElementCopyMultipleAttributeValues
This was working previously. We didn't do any significant iOS updates on the devices, so I'm not sure what could have changed.
One example is setting the device orientation:
UIDevice.shared.orientation = .portrait
When I break in the debugger at that point I also see the error when I try
po XCUIDevice.shared.orientation = .portrait.
If I try a second time then it works fine, but the first call always fails which causes the test to fail.
There are some other calls which behave similarly. First call always fails with the error above, second call is fine. They're not throwing an exception, so I can't handle it with a do/try/catch.
Is there an accessibility setting that needs to be made manually on the iOS device for this to work?
We have a need to use IPv6 literals in URLSession host addresses at times. This has its roots in issues with load balancing, and which hosts share what information, and it isn't something we can work around right now.
The IPv6 literal we're using was returned in the session metrics of a previous URLSession as the remote address of that connection.
What are the formatting requirements for that, if any? I.e., if we're setting
"61:ff4a::7a81:5002" // Not the real address :-)
As the host part of a URLComponents(), then what's sent is
61%3aff4a%3a%3a7a81%3a5002
Which is kind of what I'd expect since it's escaping it.
If we set
"[61:ff4a::7a81:5002]"
Which is what's specified in RFC-2732 then we get a host-not-found error
Is there somewhere in the documentation about how we can use an IPv6 literal as a URLSession host address?
In Supporting IPv6 DNS64/NAT64Networks the advice is "don't", but that's not one of our options...
We're seeing MPTCP traffic on the VPN's utun interface which does not match the IP routes defined in the network settings at VPN connection time.
The routes defined at connection time are all private IP address spaces (10., 192.168. etc.), but we're getting MPTCP traffic destined for some AWS addresses.
Is there any known issue with MPTCP routing & VPNs on macOS/iOS?
We've been setting DNSSettings.matchDomains to an array containing an empty string in order to have all DNS requests sent to our NEPacketTunnelProvider.
This worked fine in older versions of macOS, but we have a customer running Ventura who are seeing no DNS request being sent in that case.
Is there any known change in the behavior of DNSSettings.matchDomains in Ventura?
When I try to run unit tests in Xcode 14.3, it's often freezing up. Needs to be "Force Quit"ed.
This happens most often the first time I try to run a suite through the GUI. Running a single test sometimes freezes, but not always.
I can't pause the test execution, and when I try to exit Xcode it asks if I want to cancel the tests, but then it doesn't cancel them and I need to force quit at that point.
Is there any know issue with 14.3 that would cause this?
I used to be able to use the Console on macOS to examine iOS device issues, and identify system issues that were causing problems with my app.
The volume of logs generated with more recent versions of iOS is so large that the logs are basically useless. If I'm using Console to stream the log on macOS then it holds no more than a few seconds of data before is starts throwing away old lines.
If I use
sudo log collect --size <size>m --device-udid <UDID>
I can specify a max of 999m, and that runs out in 5 seconds on average. Can't specify 'g' as an option for size :-(
There doesn't appear to be a way to exclude logging information I don't need or to stream the log to a file with more than a 999 MB capacity.
Is there some way to do this?
I have a hard time believing that all this logging would be happening with no effective way to access it over longer times. Or maybe someone did just really mess up the logging system with current releases...
We're seeing two odd things with DNS in our Network Extension.
One is a problem which appears to be related to an interaction between enforceRoutes and DNS
One of the configurations we have is setting ourselves as the default DNS resolver by setting matchDomains to [@""], and specifying a list of DNS searchDomains.
When enforceRoutes is YES we aren't seeing all the DNS resolution we expect. When enforceRoutes is NO it looks like it's working. Same iOS version (16.3), same VPN extension. The only difference in configuration is enforceRoutes set to YES in the failing case. excludeLocalNetworks in YES in each case.
Is there some known interaction there that I'm missing?
The other issue is that some customers with that configuration aren't seeing DNS name completion. I.e., if matchDomains is
[ @"aaa.com", @"aab.com" ]
then trying to resolve
foo
in a browser should attempt foo.aaa.com and foo.aab.com. We're not seeing that at all. If matchDomains is the same as searchDomains then resolution is happening fine.
We have four build agents.
2 x Intel, 8GB RAM
1 x M1, 8 GB RAM
1 x M2, 8GB RAM
The only differences in the configurations are the processor types.
On the Intel systems things work fine. On the M1/M2 systems we see the "Force Quit Applications" dialog, with the message:
"Your system has run out of application memory."
None of the processes it offers to quit are big, but checking Activity Monitor shows testmanagerd using 9+ GB (yes GIGABYTES) of Memory. It show "Real Memory" of about 1GB.
Anyone have an idea of what's going on here? All of the systems are running the most current Ventura build, and Xcode 14.3
In our PacketTunnelProvider we are seeing behavior for enforceRoutes which appears to contradict the documentation.
According to the developer documentation (my emphasis):
If this property is YES when the includeAllNetworks property is NO, the system scopes the included routes to the VPN and the excluded routes to the current primary network interface.
If we set these IPv4 settings:
IPv4Settings = {
configMethod = manual
addresses = (
172.16.1.1,
)
subnetMasks = (
255.255.255.255,
)
includedRoutes = (
{
destinationAddress = 0.0.0.0
destinationSubnetMask = 0.0.0.0
},
)
excludedRoutes = (
{
destinationAddress = 10.10.0.0
destinationSubnetMask = 255.255.255.0
},
)
overridePrimary = YES
}
Then if enforceRoutes is set to YES, then we do not see traffic for the excluded network, which is the expected behavior. If enforceRoutes is set to NO, then we do see traffic for the excluded network.
In both cases includeAllNetworks and excludeLocalNetworks are both NO.
The excluded network is not one of the local LANs.
Is this a known issue? Is there some documented interaction that I missed here?
Is there a workaround we can use to make this function as intended, with enforceRoutes set to YES?
We're seeing server trust failures with iOS 17 that we don't see with iOS 16, particularly in debugging, when we build with Xcode 15. We handle
func urlSession(_ session: URLSession,
didReceive challenge: URLAuthenticationChallenge,
completionHandler: @escaping (URLSession.AuthChallengeDisposition, URLCredential?) -> Void)
and choose to deal with recoverable server trust failures of particular kinds ourselves. After our checks we end up calling the completion handler:
let credential = URLCredential(trust: serverTrust)
completionHandler(.useCredential, credential)
And everything continues.
This is working on macOS Venture and earlier and iOS 16 and earlier. It also works if we install our current release build on iOS 17. If we build with Xcode 15 and test on iOS 17 then calling the completion handler with .useCredential ends up failing with a -1200 error, and a message about a recoverable trust failure.
Has anyone else seen this behavior? Does anyone know if this is related to just Xcode 15, or to Xcode 15 + an interaction with iOS 17? Maybe the SDKs used with Xcode 15 are being stricter? In any case it would seem that saying .useCredential should cause it to .use the credential...
We're seeing this message when trying to run test automation on iOS 17.
We're building the app on Sonoma, using Xcode 15, and running on iOS 17 (physical iPhone 13). At one point in the test we want to verify that something is correctly set in the iOS Setting app.
We were previously doing this by launching com.apple.Preferences
On iOS 17 we're getting a failure because
"XCTAutomationSupport couldn’t be loaded because it is damaged or missing necessary resources"
I suspect that the error message is not actually representative of the real error
Anyone seen this before?
We have an App Group defined in our entitlements file so that two pieces of our software, a management GUI and a VPN extension, can write files to the same location.
This is not just for regular log files. There's data we want to record which isn't appropriate for the system logs.
What we're seeing on the iOS and macOS betas is that the write() file always fails, and we end up with \0s written to the file instead of the data.
This is true both with the shipping versions of our applications on the App Store and with builds made with Xcode 15 and run on the devices in the debugger.
Happens from both the Network Extension and the management application.
Both macOS and iOS.
Shipping apps and freshly built with latest tools.
There's nothing we see in the Console logs that would appear to explain this.
I didn't see anything in the release notes that would address this, but I could easily have missed something.
Anyone else seen this?