I found this when looking for additional information on mmap on arm64, as I have an application where mmap is failing as well. I ran your test application on Big Sur 11.3 and it currently fails completely.
cc -m64 -g -Wall -Werror -o mapfail main.c
codesign -vvv mapfail
mapfail: valid on disk
mapfail: satisfies its Designated Requirement
./mapfail 10000000 10000
maping from 0x10000000 to 0x10010000 is OK
Unable to reserve at 65536 (0x10000) bytes of memory for the test1 heap
test1 not mapped
I suspected this probably has to do with PROT_EXEC, so I tried mmap with different or'd bits.
if (base == 0 || ok_to_map(base, base+size)) {
res = (char *) mmap((void *) base, size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | FIMAP_ANON | (base ? MAP_FIXED : 0),
bucket_o_zeros, 0);
}
Removing the PROT_EXEC bit produces:
./mapfail 10000000 10000
maping from 0x10000000 to 0x10010000 is OK
Unable to reserve 0x10000000 for the test1 heap,
using 0x1040ec000 instead
test1 heap mapped from 0x1040ec000 to 0x104100000
Unfortunately for me, I need the exec bit. I'm assuming this is an attempt at further sandboxing the ecosystem, and if there's something I'm missing I'd be happy to be enlightened.