We're having a similar issue in our project, it seems that the swift libraries are included twice in a production ipa created from Xcode v7.3.1, the resulting ipa file increased from 23mb to 55mb.
Even if most of the binary isn't downloaded to the devices I'd still love to understand why the libraries are included twice, and are actually different in size.
Googling resulted in this stackoverflow thread (which is probably related to the original post) but without an answer.
➜ prosper_daily_swift.ipa ls -alh SwiftSupport/iphoneos/
total 100304
drwxr-xr-x 13 guti wheel 442B Jul 4 09:50 .
drwxr-xr-x 3 guti wheel 102B Jul 4 09:50 ..
-rwxr-xr-x 1 guti wheel 256K Apr 27 03:02 libswiftContacts.dylib
-rwxr-xr-x 1 guti wheel 41M Apr 27 03:02 libswiftCore.dylib
-rwxr-xr-x 1 guti wheel 780K Apr 27 03:02 libswiftCoreGraphics.dylib
-rwxr-xr-x 1 guti wheel 250K Apr 27 03:02 libswiftCoreImage.dylib
-rwxr-xr-x 1 guti wheel 256K Apr 27 03:02 libswiftCoreLocation.dylib
-rwxr-xr-x 1 guti wheel 532K Apr 27 03:02 libswiftDarwin.dylib
-rwxr-xr-x 1 guti wheel 336K Apr 27 03:02 libswiftDispatch.dylib
-rwxr-xr-x 1 guti wheel 4.5M Apr 27 03:02 libswiftFoundation.dylib
-rwxr-xr-x 1 guti wheel 448K Apr 27 03:02 libswiftObjectiveC.dylib
-rwxr-xr-x 1 guti wheel 337K Apr 27 03:02 libswiftPassKit.dylib
-rwxr-xr-x 1 guti wheel 540K Apr 27 03:02 libswiftUIKit.dylib
➜ prosper_daily_swift.ipa ls -alh Payload/Prosper\ Daily.app/Frameworks/
total 33832
drwxr-xr-x 13 guti wheel 442B Jul 4 09:50 .
drwxr-xr-x 1132 guti wheel 38K Jul 4 09:50 ..
-rwxr-xr-x 1 guti wheel 241K Jul 4 09:50 libswiftContacts.dylib
-rwxr-xr-x 1 guti wheel 12M Jul 4 09:50 libswiftCore.dylib
-rwxr-xr-x 1 guti wheel 397K Jul 4 09:50 libswiftCoreGraphics.dylib
-rwxr-xr-x 1 guti wheel 203K Jul 4 09:50 libswiftCoreImage.dylib
-rwxr-xr-x 1 guti wheel 241K Jul 4 09:50 libswiftCoreLocation.dylib
-rwxr-xr-x 1 guti wheel 293K Jul 4 09:50 libswiftDarwin.dylib
-rwxr-xr-x 1 guti wheel 241K Jul 4 09:50 libswiftDispatch.dylib
-rwxr-xr-x 1 guti wheel 1.6M Jul 4 09:50 libswiftFoundation.dylib
-rwxr-xr-x 1 guti wheel 289K Jul 4 09:50 libswiftObjectiveC.dylib
-rwxr-xr-x 1 guti wheel 242K Jul 4 09:50 libswiftPassKit.dylib
-rwxr-xr-x 1 guti wheel 300K Jul 4 09:50 libswiftUIKit.dylib