The memory limit for a ReplayKit upload extension is pretty small, and there's nothing you can do to increase it.
>> I have to do a lot work for video and video processing
What APIs are you using for this, or is it all done "manually"? I would expect that Video Toolbox is the only framework that has a chance of succeeding in the extension environment. If you can't use that, then your only choice might be to do the encoding on the server.
You can also try dropping some frames, and see if that helps.
Is there any chance apple will increase this memory limit in future?
50 MB iOS 12.2