16 Replies
      Latest reply on Oct 26, 2019 9:10 PM by kandre
      jedisct1 Level 1 Level 1 (0 points)

        There seems to be a regression in Xcode 11 beta, when AVX is enabled. The C library doesn't make the same assertions regarding alignment as the compiler.

         

        Here is a C-reduced test case:

         

        #include 
        #include 
        #include 
        #include 
        #include 
        
        typedef struct {
            char  d[16];
            void *e;
            struct { char b[5536]; } f;
        } i;
        
        void g(void) {
            struct addrinfo hints, *k;
            memset(&hints, 0, sizeof hints);
            getaddrinfo(NULL, NULL, &hints, &k);
        }
        
        int main(void) {
            puts("Hello world");
            fflush(stdout);
            close(open("/dev/null", O_RDONLY));
            i context;
            context.e = open;
            printf("%p\n", context.d);
            g();
            return 0;
        }

         

        When compiled with AVX optimizations (e.g. -mavx or more commonly -march=native), this crashes even before `main()` is executed:

         

        cc -mavx -O2 a.c && ./a.out
        lldb ./a.out
        run
          * thread #1, queue = 'com.apple.main-thread', stop reason = EXC_BAD_ACCESS (code=EXC_I386_GPFLT)
        frame #0: 0x00007fff65e73316 libdyld.dylib`stack_not_16_byte_aligned_error libdyld.dylib`stack_not_16_byte_aligned_error:
        ->  0x7fff65e73316 <+0>: movdqa %xmm0, (%rsp)

         

        Without AVX optimizations, or with Xcode 10, this doesn't happen.

         

        This bug affects real-world applications such as libsodium and dsvpn.

         

        A workaround is to use `-ffreestanding`.

        • Re: Xcode 11b6: stack_not_16_byte_aligned_error with AVX code
          jedisct1 Level 1 Level 1 (0 points)

          The headers got mangled. Not the most important thing, but anyway, here they are:

           

          #include <fcntl.h>
          #include <netdb.h>
          #include <stdio.h>
          #include <string.h>
          #include <unistd.h>
          
          • Re: Xcode 11b6: stack_not_16_byte_aligned_error with AVX code
            snowcat Level 1 Level 1 (0 points)

            Unfortunate combination:

            • -mavx
            • -O2
            • -fstack-check (enabled by default when building for 10.15)

             

            Here's the start of main with all these options:

            a.out[0x100000e34] <+4>:   pushq  %rbx
            a.out[0x100000e35] <+5>:   andq   $-0x20, %rsp
            a.out[0x100000e39] <+9>:   pushq  %rax
            a.out[0x100000e3a] <+10>:  movl   $0x1640, %eax             ; imm = 0x1640 
            a.out[0x100000e3f] <+15>:  callq  0x100000ef6               ; symbol stub for: ___chkstk_darwin
            a.out[0x100000e44] <+20>:  subq   %rax, %rsp
            a.out[0x100000e47] <+23>:  popq   %rax

             

            The andq is only there with -mavx and -O2. This aligns the stack to 16.

             

            The call to ___chkstk_darwin is only there with the stack check feature enabled. For specifically this call, the stack is aligned to 8 if it was previously aligned to 16. This is where the error occurs.

             

            Side note: I'm not super confident in how much this stuff is being tested. I filed a bug a few weeks ago that your test case actually also triggers. If you build like this you get a different crash:

             

               cc -all_load a.c && ./a.out

             

            (This links in the compiler runtime that ships with Xcode, which has a different ___chkstk_darwin that modifies rax, and you can see that the code I quoted above doesn't expect that.)

            • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
              jedisct1 Level 1 Level 1 (0 points)

              This critical bug is still present in XCode 11 GM.

                • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                  jcookie Level 2 Level 2 (70 points)

                  I have reported this issue as early as 8. August, with no reply from Apple. I am a bit alarmed that a bug as critical as this has been seemingly ignored for over a month. This essentially makes C/C++ compeltely useless for any non-trivial numerical code.

                    • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                      tguillem Level 1 Level 1 (0 points)

                      Hello, I don't know if it's related but a simple

                       

                      void *buf = aligned_alloc(32, 128);

                      cause a SIGABORT with the following stack strace.

                       

                      * thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGABRT

                          frame #0: 0x0000000104a22418 dyld`__abort_with_payload + 8

                          frame #1: 0x0000000104a21a58 dyld`abort_with_payload_wrapper_internal + 100

                          frame #2: 0x0000000104a21a88 dyld`abort_with_payload + 12

                          frame #3: 0x00000001049e48f8 dyld`dyld::halt(char const*) + 304

                          frame #4: 0x00000001049e4a14 dyld`dyld::fastBindLazySymbol(ImageLoader**, unsigned long) + 284

                          frame #5: 0x00000001db73f848 libdyld.dylib`dyld_stub_binder + 60

                        * frame #6: 0x000000010490e1fc alignedalloc`main(argc=1, argv=0x000000016b4f78d0) at main.m:13:20

                          frame #7: 0x00000001db73d8e0 libdyld.dylib`start + 4

                      • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                        snowcat Level 1 Level 1 (0 points)

                        You're not wrong, but there are workarounds in case you (or someone else finding this thread) just really need to make it work

                         

                        • add -fno-stack-check to CFLAGS/CXXFLAGS, or
                        • set MACOSX_DEPLOYMENT_TARGET=10.14 in the environment or choose 10.14 as deployment target in Xcode
                          • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                            jcookie Level 2 Level 2 (70 points)

                            True, and I understand that beta software can have bugs that require workarounds like this.

                             

                            But we are talkign about Xcode GM now. Imagine the mess if this bug is present in the official Catalina release. For instance, we have academic personell (non-programmers) who heavily rely on open-source numerical software, and its a common practice to build this from source using homebrew (to make sure that latest CPU capabilities are properly taken care of). These people wouldn't know why their software is randomly crashign and they don't understand compiler flags or how to look for solutions. All they know is how to run "brew install --build-from-source X" and they expect it to work.

                              • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                                jedisct1 Level 1 Level 1 (0 points)

                                I write cryptography libraries, and at this point, all I can tell to users is to avoid Xcode 11 at all costs.

                                 

                                Disabling stack checking may appear to work around the bug.

                                 

                                But the presence of such a bug suggests the Xcode 11 hasn't received enough testing before being tagged GM, which doesn't inspire confidence for compiling code that manipulates sensitive data.

                                • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                                  snowcat Level 1 Level 1 (0 points)

                                  Well the GM build shouldn't really be considered a beta anymore. A GM is a build that can reasonably be expected to not need further changes for release. I half expect it to release in this state too because of iOS 13. They might decide to fix it post release for Catalina, since there's more time until Catalina comes out.

                                   

                                  I only posted the workaround for other people in a similar position as the one I am in, which is one where I can't just suggest to wait with macOS 10.15 and Xcode 11. My job is either making stuff work, or coming up with a much more detailed explanation than "it crashes". The workaround is not for people who can just decide to not use Xcode 11. Thankfully, they've at least fixed the other bug with the stack check feature I referenced earlier in the thread.

                                   

                                  This is not a Catalina problem btw, it's Xcode 11. You can reproduce this bug on macOS 10.14 too with the Xcode GM build, but then you have to turn the feature on yourself with -fstack-check. The difference is that if you use Xcode 11 on macOS 10.15, it'll default to building for 10.15, and that turns this feature on automatically.

                                   

                                  The feature is also present in Xcode 10. But with Xcode 10, the stack stecking code is embedded in the compiled binary. Meaning the call doesn't go through the dynamic linker, which in turn means the 16 byte alignment requirement of the macOS ABI isn't enforced. So this is not like it's some major new thing in Xcode 11. It's the old combination of unfortunate circumstances.

                          • Re: Xcode 11 GM: stack_not_16_byte_aligned_error with AVX code
                            jedisct1 Level 1 Level 1 (0 points)

                            Still not fixed in Xcode 11 final.

                              • Re: Xcode 11: stack_not_16_byte_aligned_error with AVX code
                                Kevindb Level 1 Level 1 (0 points)

                                Just wanted to say thanks for the hints.  I encountered this strange behaviour when trying to build a linux kernel on Catalina, a simple line based 'make config' was giving segmentation violation.  Previously it had all 'just worked' on High Sierra.   Adding "export MACOSX_DEPLOYMENT_TARGET=10.14" to my environment variables solved.  Replying here in case others fall over this problem.

                                 

                                kevin@Kevins-MBP:~/git/github/linux (linux-5.3.y|)$ make V=1 config

                                /Applications/Xcode.app/Contents/Developer/usr/bin/make -C /Users/kevin/git/github/linux -f /Users/kevin/git/github/linux/Makefile config

                                /Applications/Xcode.app/Contents/Developer/usr/bin/make -f ./scripts/Makefile.build obj=scripts/basic

                                  gcc -Wp,-MD,scripts/basic/.fixdep.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89         -o scripts/basic/fixdep scripts/basic/fixdep.c

                                rm -f .tmp_quiet_recordmcount

                                /Applications/Xcode.app/Contents/Developer/usr/bin/make -f ./scripts/Makefile.build obj=scripts/kconfig config

                                  gcc -Wp,-MD,scripts/kconfig/.conf.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89       -c -o scripts/kconfig/conf.o scripts/kconfig/conf.c

                                  gcc -Wp,-MD,scripts/kconfig/.confdata.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89       -c -o scripts/kconfig/confdata.o scripts/kconfig/confdata.c

                                  gcc -Wp,-MD,scripts/kconfig/.expr.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89       -c -o scripts/kconfig/expr.o scripts/kconfig/expr.c

                                  flex -oscripts/kconfig/lexer.lex.c -L scripts/kconfig/lexer.l

                                  bison -o/dev/null --defines=scripts/kconfig/parser.tab.h -t -l scripts/kconfig/parser.y

                                  gcc -Wp,-MD,scripts/kconfig/.lexer.lex.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89      -I ./scripts/kconfig -c -o scripts/kconfig/lexer.lex.o scripts/kconfig/lexer.lex.c

                                  bison -oscripts/kconfig/parser.tab.c -t -l scripts/kconfig/parser.y

                                  gcc -Wp,-MD,scripts/kconfig/.parser.tab.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89      -I ./scripts/kconfig -c -o scripts/kconfig/parser.tab.o scripts/kconfig/parser.tab.c

                                  gcc -Wp,-MD,scripts/kconfig/.preprocess.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89       -c -o scripts/kconfig/preprocess.o scripts/kconfig/preprocess.c

                                  gcc -Wp,-MD,scripts/kconfig/.symbol.o.d -Wall -Wmissing-prototypes -Wstrict-prototypes -O2 -fomit-frame-pointer -std=gnu89       -c -o scripts/kconfig/symbol.o scripts/kconfig/symbol.c

                                  gcc   -o scripts/kconfig/conf scripts/kconfig/conf.o scripts/kconfig/confdata.o scripts/kconfig/expr.o scripts/kconfig/lexer.lex.o scripts/kconfig/parser.tab.o scripts/kconfig/preprocess.o scripts/kconfig/symbol.o

                                scripts/kconfig/conf  --oldaskconfig Kconfig

                                Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/4.2.1

                                make[2]: *** [config] Segmentation fault: 11

                                make[1]: *** [config] Error 2

                                make: *** [sub-make] Error 2

                                 

                                kevin@Kevins-MBP:/cores $ lldb -c core.64998

                                (lldb) target create --core "core.64998"

                                Core file '/cores/core.64998' (x86_64) was loaded.

                                (lldb) bt

                                * thread #1, stop reason = signal SIGSTOP

                                  * frame #0: 0x00007fff73f91476 libdyld.dylib`stack_not_16_byte_aligned_error

                                  • Re: Xcode 11: stack_not_16_byte_aligned_error with AVX code
                                    kandre Level 1 Level 1 (0 points)

                                    i just want to say that this has be EXTREMELY helpfull

                                     

                                    thank you @snowcat @Kevindb

                                     

                                    -------------------

                                     

                                    additional background in my case:

                                     

                                    i was attempting to build linux headers with GCC-9 and was getting the same problem as with xcode (gcc-9 was built with xcode 11 command line tools).... the solution was to build with normal xcode and set `MACOSX_DEPLOYMENT_TARGET=10.14` to build linux headers properly

                                     

                                    note:

                                    i also tried building GCC-8 with `8.3.0-xcode-bug-_Atomic-fix.patch` and see if i can still use a GCC-based toolchain... ill update here if gcc-8 is a valid workaround.....

                                • Re: Xcode 11: stack_not_16_byte_aligned_error with AVX code
                                  mouse008 Level 1 Level 1 (0 points)

                                  This problem appears to persist through Xcode-11.1 public release, and present in Xcode-11.2 (11B41) on macOS Catalina 10.15.1 (19B68f).

                                   

                                  See https://trac.macports.org/ticket/58776