Xcode is an absolute dog to install, it's heavy and slow regardless, but i had this exact same issue on a VirtualBox, i left it for about 4 hours and it made near zero progress.
I juiced up the VM ram from 4gb to 16gb, and it installed in about an hour for me. It seems clear RAM is the key, and with 16gb+ on a proper mac i imagine you'd get this under 30mins.
Always sucks when hardware is the issue.