Rendering Core Image to MTLTexture causes huge memory use

Question

Created Jan ’22

Replies 3

Boosts 0

Views 1.2k

Participants 3

I have a project where I capture live video from the camera then pass it through a chain of CIFilters and render the result into an MTLTexture. It is all working well except that each time a call to CIContext's render toMTLTexture function is called the memory usage increases by ~150Mb and never goes back down. This causes the app to be killed by the OS after about 15-20 images are processed due to memory issues.

I have isolated the issue to the following process image function:

// Initialise required filters
CIFilter *grayScaleFilter = [CIFilter filterWithName:@"CIColorMatrix" keysAndValues: @"inputRVector", [CIVector vectorWithX:1 / 3.0 Y:1 / 3.0 Z:1 / 3.0 W:0], nil];
CIFilter *blurFilter = [CIFilter filterWithName:@"CIBoxBlur" keysAndValues:kCIInputRadiusKey, [NSNumber numberWithFloat:3.0], nil];

const CGFloat dxFilterValues[9] = { 1, 0, -1, 2, 0, -2, 1, 0, -1};
CIFilter *dxFilter = [CIFilter filterWithName:@"CIConvolution3X3" keysAndValues:kCIInputWeightsKey, [CIVector vectorWithValues:dxFilterValues count:9], nil];

const CGFloat dyFilterValues[9] = { 1, 2, 1, 0, 0, 0, -1, -2, -1};
    CIFilter *dyFilter = [CIFilter filterWithName:@"CIConvolution3X3" keysAndValues:kCIInputWeightsKey, [CIVector vectorWithValues:dyFilterValues count:9], nil];

// Phase filter is my custom filter implemented with a Metal Kernel
CIFilter *phaseFilter = [CIFilter filterWithName:@"PhaseFilter"];

// Apply filter chain to input image
[grayScaleFilter setValue:image forKey:kCIInputImageKey];
[blurFilter setValue:grayScaleFilter.outputImage forKey:kCIInputImageKey];
[dxFilter setValue:blurFilter.outputImage forKey:kCIInputImageKey];
[dyFilter setValue:blurFilter.outputImage forKey:kCIInputImageKey];
[phaseFilter setValue:multiplierFilterDx.outputImage forKey:@"inputX"];
[phaseFilter setValue:multiplierFilterDy.outputImage forKey:@"inputY"];
 
// Initialize MTLTextures
MTLTextureDescriptor* desc = [MTLTextureDescriptor texture2DDescriptorWithPixelFormat:MTLPixelFormatR8Unorm width:720 height:1280 mipmapped:NO];
desc.usage = MTLTextureUsageShaderWrite | MTLTextureUsageShaderRead;
id<MTLTexture> phaseTexture = [CoreImageOperations::device newTextureWithDescriptor:descriptor];
 
// Render to MTLTexture
CGColorSpaceRef colorSpace = CGColorSpaceCreateDeviceRGB();
// Memory usage increases by ~150Mb after the following function call!!!
[context render:phaseFilter.outputImage toMTLTexture:phaseTexture commandBuffer:commandBuffer bounds:phaseFilter.outputImage.extent colorSpace:colorSpace];
    
CFRelease(colorSpace);
    
return phaseTexture;

I profiled the memory usage using instruments and found that most of the memory was being used by IOSurface objects with CoreImage listed as the responsible library and CreateCachedSurface as the responsible caller. (See screenshot below)

screen_shot_2022-01-13_at_4.57.05_pm.png

This is very strange because I set up my CIContext to not cache intermediates witht the following line:

const CIContext *context = [CIContext contextWithMTLCommandQueue:commandQueue options:@{ kCIContextWorkingFormat: [NSNumber numberWithInt:kCIFormatRGBAf], kCIContextCacheIntermediates: @NO, kCIContextName: @"Image Processor" }];

Any thoughts or advice would be greatly appreciated!

Boost

Answer 1

FrankSchlegel OP

Jan ’22

Hmm, your code seems ok so far. I guess the issue is caused by something else... Are you sure the texture you create here is released properly? Also, can you please try passing nil as commandBuffer in [context render:...]?

0

Answer 2

DavidTober OP

Jan ’22

Thanks for the comment! The texture seems to be released properly via Automatic Reference Counting. Setting the command buffer to nil does seem to fix the memory problem!! 😊🎉

So I guess the command buffer is holding onto the references filling up the memory with a heap of intermediate render values. Is there any way to tell the command buffer to release these or is setting to nil the best solution to go for?

0

Answer 3

Graphics and Games Engineer OP

Apple

Jan ’22

If you use [MTLCommandQueue commandBufferWithUnretainedReferences], encoders from the command buffer will not increase the refcount when you set the resource. Using this method just shifts the responsibility of resource tracking from the API to you. So you need to really careful here; if you release a resource and Metal (or the underlying GPU) references by executing commands, you can easily encounter a SEGFAULT or some other serious issue.

commandBufferWithUnretainedReferences isn't really designed to reduce memory cost. It's really there to reduce the relatively small CPU cost incurred by encoders when it counts references. If you know that a resource will be around until f the command buffers completes, then it makes sense to use this. For instance, a game may use it for a static texture would be needed throughout a level and will free well after the last command buffer is complete.

Really you don't if the command buffer holds on to a resource, it probably means it's really needed.

0