Looks like this is still a problem with Spirv-Cross not placing interpolation modifiers onto vertex shader outputs. Can we get any clarification from Apple, since the docs are super confusing on these. It states that only the "fragment input" needs these modifiers, but the vertex output is also mostly the fragment input. I assume the vs output and ps input both need the same modifier, or the render pipeline fails to link. https://github.com/KhronosGroup/SPIRV-Cross/issues/1542

In both cases, the capture was enabled on macOS and iOS. But only iOS listed a 500mb increase when the capture was performed in the Memory Viewer of gpu capture. The macOS numbers were reasonable, and what I expected. When I disabled gpu capture, the memory difference reported by Xcode's memory panel (not gpu capture) was only 10MB or so. We have our own reporting system, so I'll dump the numbers with and without capture enabled below. I was seeing numbers similar to this when vertex buffers weren't de-duped before being added up. That happens when you have sub-ranges of vertices/indices all packed into a single VB/IB. With and without capture enabled, the amount reported is the same by our tool that accumulated the de-duped data: Total device size: 865 MB Total vertex size: 588 MB Total texture size: 198 MB

It looks like on buffers, all of our iOS buffers are allocated to 128K minimum for even small uniform buffers (as reported by MTLBuffer.allocatedSize). device.currentAllocatedSize also reflects this larger total. But on macOS with the same Metal code, the buffers are 4K or less. This made this hard to track down, since it was 4300 small MTLBuffers x 128K = 530MB. Is there some large page table present on iPadOS that is killing us here that isn't the case on macOS (Intel). So page sizes are 4x what we see on macOS. That does waste memory. It’s 16K, not 128K though. But still if a MTLResource is mapped to a page minimum, then this does waste more space on iOS. iPadOS expr (int)getpagesize() (int) $1 = 16384 macOS expr (int)getpagesize() (int) $0 = 4096 Also this is strange, some of our buffers < 128K allocated, but the super small ones always return 128K. It's like Metal is sticking super tiny allocations into a shared buffer, and then we (and gpu capture) are double-counting the memory. VertexData1 request:0.046 actual:0.047 MB VertexData0 request:0.000 actual:0.125 MB VertexData_2 request:0.000 actual:0.125

We are using this number to sanity check our buffer and texture usage against the internal numbers that we have. For example, on macOS 3D textures were 8x bigger on non-pow2 textures than what the dimensions would suggest (384x384x3 texture). The texture correspondence looked reasonable on macOS and iOS (maybe 200K off from our totals). The buffers on iOS were way off (totaling up de-duped allocatedSize), but then device.currentAllocatedSize also reflected this much larger size that indicated that it was totaling up these larger values. We aren't hitting memory limits on this test, since my device has 4GB and we're around 0.8GB. In the past, though, we have had small buffer allocations (one quad per buffer) report 1gb of buffer memory use so we ended up optimizing it to consolidate those. Also gpu capture is reporting incorrect memory totals as a result of this so validation has to be done on macOS. And macOS reports 220MB less than iOS even in the Xcode memory totals. Before we try to consolidate to fewer MTLBuffer, having allocatedSize/currentAllocatedSize be correct on iOS would really help.

Here's the Feedback number. FB8934899 (MTLBuffer.allocatedSize and MTLDevice.currentAllocatedSize return 128K master buffer size)

My workaround for now is to see if allocatedSize returns 128*1024, and if so then use buffer.length. This gives reasonable numbers even if they're not exactly what the system allocated. Also we don't use the device.currentAllocatedSize since it's unreliable with these buffers overcounted.

Here's one theoretical workaround, but which seems like a hack to what is a fundamental dyld problem. Dynamic libraries make little sense if you can't reload them dynamically. The dyld cache has been around forever, but seems to be not functioning properly here. The modstamp on the dylib is completely different in this case, and should be detected and that version loaded, instead of returning the old dylib. Also how do you tell with otool if a dylib doesn't meet the criteria for hotloading? Are there any parameters which state that it uses ObjC, thread_local, or Swift code? There must be criteria that dyld uses to prevent hotloading. Also is there a way to tell with all of the dyld environment flags the modstamp of the dylib that was just loaded? In this case, I'm just modifying a print statement, and then running that code, but it doesn't print the changed line until the app is reloaded. if (modstamp differs from last load) {  Create a temp file (writeable since it’s in temp directory), use mkstmp  Store the modstamp  Copy newly built file from Library/Caches over to temp  dlclose() old file  dlopen() temp file dylib (does that work?, @rpath issues)  Let the system delete the temp file when user quits, next load starts from Library/Caches directory.  }

Hi Quinn, The "image list" is a good suggestion. I was just using that other day. I'll try that, and see what lldb reports. This is pure C++, no Swift or ObjC usage. For some of our users, the dylib hotloads and for others not. It's a mix of people mostly on 10.15.7 and XCode 12.2/3. I did the implementation mentioned above of renaming the lib, but it seems like we have a partial reload. So some code hotloads fine, and other code is still pointing to the old lib. Some code at init of the dylib where we crash (with renaming) is reporting the previous numbered dylib in the debugger. This may be an artifact of how Xcode loads the sources once at launch, but when I do hotload, my breakpoints show the new sources. By ODR, do you mean on-demand-resource? We did have those a while back, but I think they're disabled currently. it’s definitely not there after dlclose() on my system. But my system works. Others don't. So I'll get some more info on the machines that don't hotload. after dlopen() (lldb) image list  [623] 1BA9F1FE-153F-3F2B-94ED-A7F7CDD466D7 0x0000000180800000 foosigned.dylib after dlclose()  dyld: unloaded: <1BA9F1FE-153F-3F2B-94ED-A7F7CDD466D7> foosigned.dylib (lldb) image list  (lldb)

So here's a capture from dlopen/dlclose() of the same library renumbered at the end. The path doesn't change, and it's looks like the cache restores the original 000.dylib instead of the 001.dylib that dlopen() requested on the second call. I have dyld environment settings providing a little more insight here. (lldb) image list -b [621] other.dylib [622] other2.dylib dlopen("/foo/bar_signed_000.dylib") dyld: Mapping /foo/bar_signed_000.dylib dyld: loaded: <A6723547-512E-30A0-9EE5-6E17DB08F79B> /foo/bar_signed_000.dylib (lldb) image list -b [621] other.dylib [622] other2.dylib [623] bar_signed_000.dylib <- correct, it’s loaded and in the list dlclose() dyld: unloaded: <A6723547-512E-30A0-9EE5-6E17DB08F79B> /foo/bar_signed_000.dylib <- correct, unloaded dlopen("/foo/bar_signed_001.dylib") dyld: loaded: <A6723547-512E-30A0-9EE5-6E17DB08F79B> /foo/bar_signed_001.dylib <- dyld reused the UUID? (lldb) image list -b [627] bar_signed_000.dylib <- ugh, this is wrong path

I should add that in the lldb session above, the library hotloads correctly despite "image list" reporting the name of the older library. Also "image list barsigned000.dylib" always reports something even when the library isn't loaded, but "image list" does not. It would be nice to not have to list every dylib from lldb to verify where one is loaded, but that appears to not be possible with current lldb. Also I forgot it in the example, but "image list -b" no longer listed the barsigned000.dylib after dlclose(). So that in addition to the dyld printout leads me to believe the dylib was purged. So we get stuck in a state where hotloading stops working, and we get constant crashes in dylib init code at startup after a test hotload that shouldn't crash. Then a few hours later, this problem goes away, as if there was a timeout on the dyld dylib cache that is failing.

Also just to emphasize the broken behavior. If lldb cannot report the correct data, then I'm not sure how to triage this. "image list" and "image list <name>" report different output. One doesn't and one does list the original dylib that's already been unloaded (at least according to dyld printouts). So I think the dyld gets confused, and starts trying to return the original dylib from time to tim. (lldb) image list -b -m bar_signed_000.dylib [  0] bar_signed_000.dylib Fri Jan 15 11:23:27 2021 dlclose(lib) <- "bar_signed_000.dylib" dyld: unloaded: <A6723547-512E-30A0-9EE5-6E17DB08F79B> /foo/bar_signed_000.dylib <- great dlopen(“/foo/bar_signed_001.dylib”) (lldb) image list -b -m /foo/bar_signed_000.dylib [ 0] bar_signed_000.dylib Fri Jan 15 11:23:27 2021. <- why is this still listed? it’s not in "image list" dyld: Mapping /foo/bar_signed_001.dylib dyld: loaded: <A6723547-512E-30A0-9EE5-6E17DB08F79B> /foo/bar_signed_001.dylib (lldb) image list -b -m Game_signed_001.dylib error: no modules found that match 'Game_signed_001.dylib’ <- dyld said it just loaded 001, why can “image list” find it?

Yes I also need the return address which wasn't in the email about the credit. Also how to you erase the unit? Holding power or cmd+R doesn't seem to work on the M1 macs.

This was the CMake command to build with Rez. Finally got it to generate a non-zero file. The -useDF to use the data fork was important. I guess will move to command line Rez builds instead, but the plugin isn't recognized like it is when Xcode processes the .r file. That may have other causes. add_custom_command(TARGET ${myTargetApp} PRE_BUILD DEPENDS ${MY_SOURCE_DIR}/MyResource.r COMMAND ${rezCompiler} several .r are located across the build I ${MY_SOURCE_DIR}/resources/ arch x86_64 use the datafork useDF needs to specify for Carbon.r and CoreServices.r? #-F Carbon #-F CoreServices where to find framework files isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/ o "${MY_SOURCE_DIR}/${myTargetApp}.rsrc" ${MY_SOURCE_DIR}/MyResource.r )

The forums allow a reply and then say the content is restricted. That loses so many replies. So we think we found the issue with dyld return the same dylib on hotload and even on relaunch of the application. Unlike linux, which sets RTLD_LOCAL by default, Apple's "man dlopen" indicates that RTLD_GLOBAL is set by default on macOS. We verfied with "image list -b -m foo.dylib" that the timestamp doesn't update while running when the dylib is changed out during a hotload with the default setting (RTLD_GLOBAL) and we only set RTLD_NOW. Also relaunching the app, still returns the old dylib and no the new one in the folder. So something about the internal caching and trying to accelerate re-launch isn't correct. We now set RTLD_LOCAL | RGLD_NOW and are seeing the correct dylib behavior. The new dylib is picked up during app execution and when the app is relaunched.

We also have the same issue. This used to work in iOS 9, and was blocked in iOS 10+. Can we have a mode or entitlement that removes the code blocking mmap from an alternate folder? There are many titles that need to hotload C++ dylibs during development, and this restriction totally kills that. Our min-spec can't move off iOS 9 until this issue is addressed. Swift supports hotloading now, so this really can't be an Apple only feature.