Undocumented 'NSViewActuallyUpdateFromLayoutEngine' call adds width when light/dark mode changes

This is a fairly bizarre behavior and I wonder if anyone with deep knowledge of NSView layout internals has a clue.

I have a view that hosts a number of toolbar-like buttons, some subclasses of NSButton and other subclasses of NSPopUpButton. The latter are used in pull-down mode and do not display a title (just a pull-down indicator and icon), they are similar in size and appearance to the NSButtons.

When the app is switched between light mode and dark mode, some layout action is triggered which always adds 5 pixels of widths to the pull-down buttons. When I override setFrame and set a breakpoint, this is happening below NNAApplication nextEventMatchingMask in an undocumented function called NSViewActuallyUpdateFromLayoutEngine.

This method always adds 5 pixels to width for repeated light/dark mode changes. It does so on a pull-down even if it has already done so before, so these controls just keep growing, and this does not seem to be related to any obvious intrinsic content.

Some things I would note in trying to track it down:

• It only happens to the popup buttons, not the regular NSButtons in the same superviews
• I have some of these in a horizontal stack view and some positioned statically in a regular NSView - both scenarios get this 5-pixel-width-resize
• I've tried overriding both instrinsicContentSize and fittingSize to return the current frame size, as a way to tell the layout system not to change the size, and that didn't work
• There are no explicit constraints created for these views; they are created in code not in Interface Builder
• I've tried using the autolayout mask value NSViewNotSizable with the translates flag set to YES, that does not prevent the issue
• I've tried setting the superview autoresizesSubviews flag to false and also making sure translates flags were false to not create unwanted constraints, these also do not prevent the issue
• The only code values I had related to 5 pixels were a spacing setting and a left inset on the horizontal stack view - so I removed those as a check, but as I'd expect that did not change the behavior

The only way I've been able to defeat it is to add a .sizeLocked property to my subclass and, if set, replace the incoming frame.size with the current one to prevent the resize while still allowing origin to change. Obviously that's not ideal but as a brute-force workaround it works for this specific issue. However I'm seeing other weird layout creep as well on repeated light/dark change.

I wonder why this happens, and what the correct fix might be! Why is dark/light mode switching resizing anything in layout, when the size of windows and views have not changed?