Xcode 11: stack_not_16_byte_aligned_error with AVX code

There seems to be a regression in Xcode 11 beta, when AVX is enabled. The C library doesn't make the same assertions regarding alignment as the compiler.


Here is a C-reduced test case:


#include 
#include 
#include 
#include 
#include 

typedef struct {
    char  d[16];
    void *e;
    struct { char b[5536]; } f;
} i;

void g(void) {
    struct addrinfo hints, *k;
    memset(&hints, 0, sizeof hints);
    getaddrinfo(NULL, NULL, &hints, &k);
}

int main(void) {
    puts("Hello world");
    fflush(stdout);
    close(open("/dev/null", O_RDONLY));
    i context;
    context.e = open;
    printf("%p\n", context.d);
    g();
    return 0;
}


When compiled with AVX optimizations (e.g. -mavx or more commonly -march=native), this crashes even before `main()` is executed:


cc -mavx -O2 a.c && ./a.out
lldb ./a.out
run
* thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
frame #0: 0x00007fff65e73316 libdyld.dylib`stack_not_16_byte_aligned_error libdyld.dylib`stack_not_16_byte_aligned_error:
-> 0x7fff65e73316 <+0>: movdqa %xmm0, (%rsp)


Without AVX optimizations, or with Xcode 10, this doesn't happen.


This bug affects real-world applications such as libsodium and dsvpn.


A workaround is to use `-ffreestanding`.

Replies

The headers got mangled. Not the most important thing, but anyway, here they are:


#include <fcntl.h>
#include <netdb.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

Unfortunate combination:

  • -mavx
  • -O2
  • -fstack-check (enabled by default when building for 10.15)


Here's the start of main with all these options:

a.out[0x100000e34] <+4>:   pushq  %rbx
a.out[0x100000e35] <+5>:   andq   $-0x20, %rsp
a.out[0x100000e39] <+9>:   pushq  %rax
a.out[0x100000e3a] <+10>:  movl   $0x1640, %eax             ; imm = 0x1640 
a.out[0x100000e3f] <+15>:  callq  0x100000ef6               ; symbol stub for: ___chkstk_darwin
a.out[0x100000e44] <+20>:  subq   %rax, %rsp
a.out[0x100000e47] <+23>:  popq   %rax


The andq is only there with -mavx and -O2. This aligns the stack to 16.


The call to ___chkstk_darwin is only there with the stack check feature enabled. For specifically this call, the stack is aligned to 8 if it was previously aligned to 16. This is where the error occurs.


Side note: I'm not super confident in how much this stuff is being tested. I filed a bug a few weeks ago that your test case actually also triggers. If you build like this you get a different crash:


cc -all_load a.c && ./a.out


(This links in the compiler runtime that ships with Xcode, which has a different ___chkstk_darwin that modifies rax, and you can see that the code I quoted above doesn't expect that.)

Smaller test case:


int main(void) {
    register char a __asm("rbx") = 0;
    char b[5000];
    char c[100] = {0};
    asm volatile("" : : "r,m"(a), "r,m"(b), "r,m"(c) : "memory");
    return 0;
}

This critical bug is still present in XCode 11 GM.

I have reported this issue as early as 8. August, with no reply from Apple. I am a bit alarmed that a bug as critical as this has been seemingly ignored for over a month. This essentially makes C/C++ compeltely useless for any non-trivial numerical code.

Hello, I don't know if it's related but a simple


void *buf = aligned_alloc(32, 128);

cause a SIGABORT with the following stack strace.


* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT

frame #0: 0x0000000104a22418 dyld`__abort_with_payload + 8

frame #1: 0x0000000104a21a58 dyld`abort_with_payload_wrapper_internal + 100

frame #2: 0x0000000104a21a88 dyld`abort_with_payload + 12

frame #3: 0x00000001049e48f8 dyld`dyld::halt(char const*) + 304

frame #4: 0x00000001049e4a14 dyld`dyld::fastBindLazySymbol(ImageLoader**, unsigned long) + 284

frame #5: 0x00000001db73f848 libdyld.dylib`dyld_stub_binder + 60

* frame #6: 0x000000010490e1fc alignedalloc`main(argc=1, argv=0x000000016b4f78d0) at main.m:13:20

frame #7: 0x00000001db73d8e0 libdyld.dylib`start + 4

You're not wrong, but there are workarounds in case you (or someone else finding this thread) just really need to make it work


  • add -fno-stack-check to CFLAGS/CXXFLAGS, or
  • set MACOSX_DEPLOYMENT_TARGET=10.14 in the environment or choose 10.14 as deployment target in Xcode

True, and I understand that beta software can have bugs that require workarounds like this.


But we are talkign about Xcode GM now. Imagine the mess if this bug is present in the official Catalina release. For instance, we have academic personell (non-programmers) who heavily rely on open-source numerical software, and its a common practice to build this from source using homebrew (to make sure that latest CPU capabilities are properly taken care of). These people wouldn't know why their software is randomly crashign and they don't understand compiler flags or how to look for solutions. All they know is how to run "brew install --build-from-source X" and they expect it to work.

This is unrelated. `aligned_calloc` is a new API, that is set to `NULL` on older OS versions.

I write cryptography libraries, and at this point, all I can tell to users is to avoid Xcode 11 at all costs.


Disabling stack checking may appear to work around the bug.


But the presence of such a bug suggests the Xcode 11 hasn't received enough testing before being tagged GM, which doesn't inspire confidence for compiling code that manipulates sensitive data.

Well the GM build shouldn't really be considered a beta anymore. A GM is a build that can reasonably be expected to not need further changes for release. I half expect it to release in this state too because of iOS 13. They might decide to fix it post release for Catalina, since there's more time until Catalina comes out.


I only posted the workaround for other people in a similar position as the one I am in, which is one where I can't just suggest to wait with macOS 10.15 and Xcode 11. My job is either making stuff work, or coming up with a much more detailed explanation than "it crashes". The workaround is not for people who can just decide to not use Xcode 11. Thankfully, they've at least fixed the other bug with the stack check feature I referenced earlier in the thread.


This is not a Catalina problem btw, it's Xcode 11. You can reproduce this bug on macOS 10.14 too with the Xcode GM build, but then you have to turn the feature on yourself with -fstack-check. The difference is that if you use Xcode 11 on macOS 10.15, it'll default to building for 10.15, and that turns this feature on automatically.


The feature is also present in Xcode 10. But with Xcode 10, the stack stecking code is embedded in the compiled binary. Meaning the call doesn't go through the dynamic linker, which in turn means the 16 byte alignment requirement of the macOS ABI isn't enforced. So this is not like it's some major new thing in Xcode 11. It's the old combination of unfortunate circumstances.

Still not fixed in Xcode 11 final.

Just wanted to say thanks for the hints. I encountered this strange behaviour when trying to build a linux kernel on Catalina, a simple line based 'make config' was giving segmentation violation. Previously it had all 'just worked' on High Sierra. Adding "export MACOSX_DEPLOYMENT_TARGET=10.14" to my environment variables solved. Replying here in case others fall over this problem.


kevin@Kevins-MBP:~/git/github/linux (linux-5.3.y|✔)$ make V=1 config

/Applications/Xcode.app/Contents/Developer/usr/bin/make -C /Users/kevin/git/github/linux -f /Users/kevin/git/github/linux/Makefile config

/Applications/Xcode.app/Contents/Developer/usr/bin/make -f ./scripts/Makefile.build obj=scripts/basic

gcc -Wp,-MD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -o scripts/basic/fixdep scripts/basic/fixdep.c

rm -f .tmp_quiet_recordmcount

/Applications/Xcode.app/Contents/Developer/usr/bin/make -f ./scripts/Makefile.build obj=scripts/kconfig config

gcc -Wp,-MD,scripts/kconfig/.conf.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/kconfig/conf.o scripts/kconfig/conf.c

gcc -Wp,-MD,scripts/kconfig/.confdata.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/kconfig/confdata.o scripts/kconfig/confdata.c

gcc -Wp,-MD,scripts/kconfig/.expr.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/kconfig/expr.o scripts/kconfig/expr.c

flex -oscripts/kconfig/lexer.lex.c -L scripts/kconfig/lexer.l

bison -o/dev/null --defines=scripts/kconfig/parser.tab.h -t -l scripts/kconfig/parser.y

gcc -Wp,-MD,scripts/kconfig/.lexer.lex.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -I ./scripts/kconfig -c -o scripts/kconfig/lexer.lex.o scripts/kconfig/lexer.lex.c

bison -oscripts/kconfig/parser.tab.c -t -l scripts/kconfig/parser.y

gcc -Wp,-MD,scripts/kconfig/.parser.tab.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -I ./scripts/kconfig -c -o scripts/kconfig/parser.tab.o scripts/kconfig/parser.tab.c

gcc -Wp,-MD,scripts/kconfig/.preprocess.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/kconfig/preprocess.o scripts/kconfig/preprocess.c

gcc -Wp,-MD,scripts/kconfig/.symbol.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89 -c -o scripts/kconfig/symbol.o scripts/kconfig/symbol.c

gcc -o scripts/kconfig/conf scripts/kconfig/conf.o scripts/kconfig/confdata.o scripts/kconfig/expr.o scripts/kconfig/lexer.lex.o scripts/kconfig/parser.tab.o scripts/kconfig/preprocess.o scripts/kconfig/symbol.o

scripts/kconfig/conf --oldaskconfig Kconfig

Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1

make[2]: *** [config] Segmentation fault: 11

make[1]: *** [config] Error 2

make: *** [sub-make] Error 2


kevin@Kevins-MBP:/cores $ lldb -c core.64998

(lldb) target create --core "core.64998"

Core file '/cores/core.64998' (x86_64) was loaded.

(lldb) bt

* thread #1, stop reason = signal SIGSTOP

* frame #0: 0x00007fff73f91476 libdyld.dylib`stack_not_16_byte_aligned_error

This problem appears to persist through Xcode-11.1 public release, and present in Xcode-11.2 (11B41) on macOS Catalina 10.15.1 (19B68f).


See https://trac.macports.org/ticket/58776

Can reproduce. This has serious implications for my codes, that are currently malfunctioning throughout.