I managed to cancel the installation somehow (MacBook Pro M1 - Monterey)
First I moved Xcode to bin (not sure why it was already installed in the first place, when the install never finished and I was installing on a new machine).
Then in launchpad I pressed and held the Xcode icon, and after a while closed it with the X icon. I think I had to do it twice, but in the end I got rid of the neverending installation.