After spending annoying amount of time to figure this issue out, I was finally able to fix it for me.
This is stupid but, installing VisionOS runtime resolved the issue, for me.
This can be done in Xcode > Settings > Platforms
xCodeWarrior
Users receive reputation points by contributing quality content to the forums.
Posts include post and replies published on the forums.
Post authors can mark replies as Accepted to indicate successful solutions.
Apple Developer Forums admins can mark replies as Apple Recommended to indicate an approved solution
Post
Replies
Boosts
Views
Activity
Answering my own question here, for anyone who might face similar issue and don't get any responses:
I ended up adding libswift_Builtin_float.tbd to the Link Binary With Libraries in the Build Phases of the Target
Then setting the Status to Optional instead of Required.
After that, building the project and running it on iPadOS 17 devices, fixed the backward compatibility crash.