Solved!
Got a new MBP16" and had the same issue (Safari is slow while Chrome flys)
What I did: In safari Preference -> General, Changed the "New {window, tab} opens with" to "Empty Page"
Emptyed the cache (cmd+opt+T)
Quit and reopend Safari
That did not solve it. I also tried remove website data and reopening without success.
Eventually i did an NVRAM restart (cmd+opt+P+R) - and it solved the issue!
I'm not sure if the initial steps were essential for the fix, or if changing the "opens with" back will cause the issue again, but for now things are flying :)