I don't have Stage Manger turned on.
One thing that seems to have helped is to not have any windows in full screen mode. It seems to be the apps that are in full screen mode which cause WindowServer to crash.
BTW, I have multiple monitors - two on lightning and one HDMI.