Memory mapped file: "Cannot allocate memory"

This is not about physical memory but address space. iOS puts limits on the address space available to app processes, meaning your can’t map files beyond a certain size.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thank you for the infomation you provide, that realy make sense to me.

But how can i release the address space?

I use "UIGraphicsBeginImageContextWithOptions", and I got this "CGBitmapContextInfoCreate: unable to allocate 6512640 bytes for bitmap data",so I can't get the image I want, but the memory is just 170M~180M, and App doesn't crash, I just want the image.

Could you give me some advise?

But how can i release the address space?

By freeing the address space using the deallocator associated with the allocator you used. For example:

• If you mapped a file with

  mmap

  , you’d free the address space with

  free

  .

• If you mapped a file with

  NSData

  , you’d free the address space by releasing all of the references to the resulting data object.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

"iOS puts limits on the address space" OMG! The only advantage of 64 bit (vs 32 bit) is/was the support for up to 16 Exa-Byte memory, no system has that much "physical" memory (of course), but at least virtual-address-space should NEVER be limited to physical-address-space.

Isn't it time to fix iOS mistakes?

(Instead of telling people that mapped files are useless)

While there have been some changes in this space in the 2 years since this thread was last active [1], my initial response on this thread is still accurate in general.

Isn't it time to fix iOS mistakes?

This isn’t a “mistake” but a deliberate design choice. Keep in mind that iOS shares a kernel with macOS, and macOS behaves the way that you consider to be ‘correct’. The limits your see on iOS are, like most things in engineering, a balance between constraints.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Like the com.apple.developer.kernel.increased-memory-limit entitlement.

Just stumbled upon this in an app I'm working on. I set a symbolic breakpoint on malloc_error_break which ends up getting hit on the following line in my code:

   CGImageRef cgImage = CGImageSourceCreateThumbnailAtIndex(imageSource,

                                                             0,

                                                             (CFDictionaryRef)options);

The imageSource above is created via a CGImageSourceCreateWithURL function call.

I use this to load images in a table view. All works well, but after scrolling the table view for awhile it appears I hit the address space limit.

I don't use mmap directly and after I get CGImage's from above I covert them to UIImage and release it:

CFRelease(imageSource);
CFRelease(imageProperties);
 if (cgImage != NULL)

    {
        UIImage *uiIMage = [[UIImage alloc]initWithCGImage:cgImage];

        CGImageRelease(cgImage);

        return uiIMage;

    }

The returned UIImages are held in a NSCache object. Is it possible that CGImageSourceCreateThumbnailAtIndex is causing me to hit the address space limit? I don't appear to have a memory leak here so I'm not sure why my address space limit would keep accumulating as I continue scrolling the table view. The image source and CGImage are released after converting to a UIImage.

I was under the impression NSCache would remove objects at the appropriate time based on certain OS conditions … automatically.

This is a recurring theme on this thread, namely that logical memory (that is, your address space) and physical memory are different things. NSCache removes items when physical memory gets short. It does not monitor your address space.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Cool got it. Thanks for the reply.

I'm guessing the answer tot his question is no but I'll ask anyway. Is there any API on iOS to detect when an app's address space limit is getting close to being hit? That is, to be notified preemptively before hitting the limit so I can free some, rather than waiting for a ENOMEM to occur..and then freeing up address space?

Now I believe this address space issue is also the reason for the out-of-memory errors I’m seeing with my Swift code but I’m lacking good ideas on how to properly fix them.

I’m receiving video frames from IP cameras with FFMPEG (avformat), convert them to CVPixelBuffer using CVPixelBufferCreateWithPlanarBytes(…), then to CGImage using VTCreateCGImageFromCVPixelBuffer(…), and finally to UIImage. I keep a buffer of the latest 100-200 images for various things and discard or save the older ones.

At first, the memory usage was just rising quickly until a crash. Then I‘ve put an autoreleasepool{…} inside the loop around the frame-receiving code which kept the memory usage stable at around ~1.4 GB (with 1920x1080 video). But this still occasionally led to an out-of-memory crash. Yet, those crashes do not go away when further reducing the maximum memory usage to around 800 MB (by creating the UIImages from JPEG-compressed image Data instead of raw pixel buffers).

When I add code to catch those out-of-memory exceptions, cancel the work on that frame and just continue with the next one, I observe that it only happens for a few frames in a row and then continues without issues for a while until the exceptions come again. I do not see substantial peaks in memory usage in the debugger. (I use Xcode on my Mac to run the app on a 4K Apple TV with tvOS 15.5.1.)

Is this really an address space issue? And how would I handle it properly then?

I stumbled across this thread rather by accident, but it reminded me of a problem I have been working lately, so I thought I would describe my situation in case the details were of interest. Specifically, I seem to have run across a 2.5 GByte limit in creating mmapped files under macOS (not iOS), and I have some details about what is going on.

I was working on a macOS app using an Intel Mac Pro as development platform, running Big Sur, using the appropriate Xcode. I intended to develop an app that would run on both Macs with Intel processors and Macs with "Apple silicon" processors, but to get going I was only worrying about the Intel side of things. I wanted to mmap a large area at a virtual address that I needed to specify at compile time (I will spare you the details of why, unless someone is interested). My approach was to use .zerofill directives in a .s file that was part of the build, like this:

.zerofill __WSS, __Wss000, _WraithSpace,                0x7f000000LL

and then add entries under "Other Linker Flags" in the Xcode project settings, like this:

-segaddr __WSS 0x20000000000

That combination creates a section named __Wss000 in the BSS of my executable, of size 0x7f000000, at virtual address 0x20000000000, which is just what I intended. I confirmed its existence by invoking "objdump -h" on the Intel version of my executable (more about the Apple silicon version later) -- which is way down in the Contents/MacOS directory inside my app's package. "objdump -h" prints out (among other things):

Sections:
Idx Name             Size     VMA              Type
    [...]
30 __Wss000         7f000000 0000020000000000 BSS

I could then mmap using the known address and size.

I didn't pick the section size of hex 7f000000 by choice: I wanted a larger section, but when I used larger sizes, the Xcode build refused to put them in the executable. The output of objdump -h was vastly different -- there was a section called "HUGE" that I did not know what to do with, the section I wanted was not there, and the app didn't work. I played around with the size, noticed that the largest size that would work was something near 2.5 GByte (7f000000 hex is 2.13 GByte), and made do with that.

I suspect that I had run up against a version of the 2.5 GByte limitation mentioned at the start of this thread, but ...

NOTE TWO THINGS:

(1) This was developing an app for macOS, not iOS.

(2) The 2.5 GByte limit apparently did not exist on Apple silicon! I ended up using the "TargetConditionals.h" header (which Apple provides) and some preprocessor directives in my .s file, like this (actual code from my source files):

#import "TargetConditionals.h"


#if TARGET_CPU_X86_64

.zerofill ClowderSegment, ClowderSector, _ClowderSpace, 0x00f00000LL
.zerofill __WSS, __Wss000, _WraithSpace,                0x7f000000LL

#elif TARGET_CPU_ARM64

.zerofill ClowderSegment, ClowderSector, _ClowderSpace, 0x2000000000LL
.zerofill __WSS, __Wss000, _WraithSpace,                0x2800000000LL

#endif

This code created the small segment in the Intel executable, that I just described, but when I moved the app to the M1 Mac Mini I had bought for testing, and ran objdump -h on the executable there I found a segment 0x2800000000LL long (that's 171 GByte) at 0x2000000000, and my app can use it with no problems (there are of course some similar #ifdefs here and there in the C++ code to be sure that the two different executables each are using the correct addresses for mmapping and such.)

Oh, and there's ONE THING MORE ...

What I was actually doing was getting around to updating a rather old app to run on both kinds of processors. In fussing with the old code, I noticed that some version of the "2.5 GByte limit problem" appears to have been around for more than a decade. I have not investigated in detail and I do not remember all the details, but I have been having problems with mmapping large chunks of memory since the days of macOS 10.6 ("Snow Leopard")!

In case any of you are terminally curious, the app is a parallel processing Scheme interpreter (Scheme is a version of Lisp), and the big mmapped chunk is Scheme main memory shared between all the versions of the interpreter that are running in parallel. Google "Wraith Scheme Jay Freeman" and you will probably find it.