EndpointSecurity Process Hangs Then Dies

App & System Services Drivers
mdolan OP
Created Jan ’20
Replies 10
Boosts 0
Views 2.9k
Participants 4

Hi all (especially Quinn if you watching),


Using everyone's favorite Endpoint Security demo code to narrow down a problem I've been seeing in my code, https://gist.github.com/Omar-Ikram/8e6721d8e83a3da69b31d4c2612a68ba (thank you, Omar), I've been able to reproduce a kill of the ES process.


If I subscribe to either the ES_EVENT_TYPE_AUTH_OPEN or ES_EVENT_TYPE_AUTH_MMAP event, when I run a leaks command on the process, it hangs, then eventually dies with a Killed: 9 message. Those are the only 2 events I've found that do this, though my search hasn't been exhaustive. I am also guessing that there are other commands besides leaks that will do this, but I can reproduce this 100% of the time with leaks.


I've tried using the async dispatch to make sure I'm not totally blocking the kernel, I've tried shortcutting the code to always just return ES_AUTH_RESULT_ALLOW to es_respond_auth_result without doing anything else, I've even considered consulting a witch doctor, but thought I'd ask here first. 🙂


Is this a known issue? I hope it's not by design, this would give malware a way to kill an EndpointSecurity process so it could try to gain a foothold before the EndpointSecurity process could restart.


If this is a problem, I'll write up a RADAR ticket and post the number here. If not, what is the suggested work-around?


Thanks!

Mike

Replies  10
Boosts  0
Views  2.9k
Participants  4
DTS Engineer OP
Apple
Jan ’20

If this is a problem, I'll write up a RADAR ticket and post the number here.

I don’t know what’s going on here but this clearly warrants a bug report IMO.

Please post your bug number, just for the record.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
mdolan OP
Jan ’20

I will do that and post the number. Thanks, Quinn.

0
mdolan OP
Jan ’20

Feedback # FB7526331


First time I used the Feedback Assistant and not the RADAR webpage. Interesting...

0
mdolan OP
Jan ’20

I submitted the Feedback Assistant ticket on Wednesday, but haven't gotten any feedback yet. Should I have at least gotten a "thank you for your submission" message, even if it was an automated response?

0
DTS Engineer OP
Apple
Jan ’20

AFAIK Feedback Assistant doesn’t send auto ACK emails. Rather, you can check the status of your bug via the web site.

I looked at your bug (FB7526331) and it’s definitely landed in the right place.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware 

let myEmail = "eskimo" + "1" + "@apple.com"
0
mdolan OP
Jan ’20

Thanks, Quinn.

0
aweqrwer OP
Feb ’20

I've noticed that ES_EVENT_TYPE_AUTH_OPEN requires responding with es_respond_flags_result(4) instead of es_respond_auth_result(4). Later one always returns ES_RESPOND_RESULT_ERR_EVENT_TYPE for me and freezes my mac because the auth events are not responded properly. May that be your case also?

0
mdolan OP
Feb ’20

Unfortunately no, they work in the normal case where a file besides my running binary is being opened. And I can get it to happen on just hte mmap case, no auth open event in the event subscribed list.


Thanks for the idea though.

0
suMac OP
Mar ’21
I could extend the complaint to any kind of debugging / externally looking at / even "Sampling" a working EndpointSecurity client daemon/app.

In my experience, it doesn't matter which kind of events I'm registering for, it doesn't matter if I respond immediately (synchronously) or in deferred block,

Any attempt to "poke" into the process causes (at least here - MacOS 10.15.6) complete system hang (at least Mac UI is dead so I can't really know if anything lives at that time) for about 30-40 seconds, and then OS kills my ES client process (kill -9) and then OS is free and back to normal.

For that I completely gave up any debugger use, any Instruments use, or actually any debugging tool except os_log() messages.

I had that guess, more of a gut feeling, that "playing inside the authorisation scheme" OS doesn't like anyone to interfere, and is so phobic about it, that it prefers killing the thing. Again - just a feeling.

0
DTS Engineer OP
Apple
Mar ’21
Someone ended up opening a DTS tech support incident about this and I dug into it in that context. It turns out that many of our tools (including leaks and vmmap) have a -forkCorpse option that causes the tool to fork a corpse process from the sysex process and then run against that corpse. That means that the tool doesn’t need to suspend the sysex process, which avoids the deadlock.

AFAICT sample doesn’t support this (I just created a file called orkCorpse!) but that’d make a fine enhancement request.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"
0
EndpointSecurity Process Hangs Then Dies
First post date Last post date
 
 
Q