Hi,
It's always hard to interpret an instrument trace from a screenshot, but my quick read is basically that you're looking at the wrong data. Two points that might be helpful:
-"mach_msg_trap" is how threads block waiting for mach message. So a thread in "mach_msg_trap" isn't really "doing" anything, it's just idled in the kernel waiting for something to happen. I generally assume that any thread in blocked "mach_msg_trap" should basically be ignored unless the thread itself is special (like the main thread) or there is some indication that there is something "interesting" about that thread.
-It's easy to overlook just how INSANELY short a microsecond (µs) is. One of your threads is listed in a call to CFArrayCreateMutable for "5.01ms" which clearly sounds like "a short time". The problem is that "5.01 ms" is "5010 µs". In other words, that SINGLE self weighted time is using MORE time than all of the other time entries combined.
The place I'd start with here is the main thread, which is either:
-
Directly blocked in something "useful" (NOT mach_msg_trap), in which case you should figure out where/why it's blocked.
-
Doing "nothing", probably in mach_msg_trap. This case is trickier to debug, but that typically means that it's waiting for something else to happen. In that case, I like to focus on:
-How did it get here? What was it doing before it started waiting? What does that tell/explain about why it doesn't always hang like this?
-Why did it stop hanging (assuming the hang "clears" on it's own)? Whatever allows it to continue is obviously what it was waiting for and following that trail should lead to the underlying issue.
Kevin Elliott
DTS Engineer, CoreOS/Hardware
