11 Replies
      Latest reply on Feb 16, 2020 9:47 AM by eskimo
      Leo Braun Level 1 Level 1 (0 points)

        Hi Eskimo,

         

        Thank you for your reply on https://forums.developer.apple.com/thread/115670

         

        As you suggested, I start a new thread here.

         

         

        I finally got to the bottom of the issue (after wasting few more hours)...

         

        Here's the situation in brief:

         

        -I successfully sign my app for notarization.

         

        -I successfully notarize it with altool and successfully staple. Therefore, I now have an officially notarized app.

         

        -However, the notarization process creates an invisible ._Info.plist file in the app's Contents folder.

         

        Which results in the following outcome when checking the signature with codesign:

         

        codesign -vvv --deep --strict appPath

         

        --prepared:aFrameworkPath/Versions/Current/.

        --validated:aFrameworkPath/Versions/Current/.

        appPath: a sealed resource is missing or invalid

        file added: appPath/Contents/._Info.plist

         

        Like I mentioned, this notarized app then goes into my custom installer on dmg, which dmg I also notarize.

         

        However, the above issue with ._Info.plist file causes error 65 on attempt to staple the dmg (after successfully running altool):

         

        Stapling failed

         

        Processing: dmgPath

        CloudKit query for dmgName.dmg (2/8162101e375e6e3fd1b06cc397f56181b19db60c) failed due to "record not found".

        Could not find base64 encoded ticket in response for 2/8162101e375e6e3fd1b06cc397f56181b19db60c

        The staple and validate action failed! Error 65.

         

         

        SOLUTION:

         

        Delete the ._Info.plist file from the app's Contents folder after notarization. It eliminates the error 65 - and dmg with installer can be successfully notarized.

         

         

        CONCLUSION:

         

        There's apparently a severe bug in altool (or stapler?) that injects malicious invisible files into the package of notarized apps - which invalidate their signature.

         

        I notarize my apps successfully for months - yet this experienced bug only yesterday for the first time. Also, it so far only happens with one of my apps.

         

        I also submitted this bug to Apple via Feedback Assistant.

         

         

        Thanks,

        Leo

        • Re: altool injects malicious invisible files into notarized apps
          eskimo Apple Staff Apple Staff (12,975 points)

          However, the notarization process creates an invisible ._Info.plistfile in the app's Contents folder.

          It’s very unlikely that this is being created by the notarisation process per se, and these files are definitely not “malicious”.

          The ._ prefix indicates an Apple Double file, which is used to store Apple-specific metadata (Finder info, resources fork, extended attributes) for a file if:

          • The file is copied to a volume where the underlying volume format does not support this metadata

          • The file is placed in some sort of archive, like a zip archive, where the archive format does not support this metadata

          You shouldn’t ever see Apple Double files on Apple volumes, like APFS and HFS Plus.  And that leads to my first question: Are you doing this work on an Apple volume?  I’d like you to check both your main work volume and the volume inside the disk image.

          Also, what does the Apple Double file contain?  That might give you a hint as to how it got created.  If the file is small, you can post a hex dump here and I’ll take a look.

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: altool injects malicious invisible files into notarized apps
              Leo Braun Level 1 Level 1 (0 points)

              Quinn,

               

              Thanks for the detailed reply.

               

              First, I ran another test and here's more accurate info:

               

              The ._Info.plist file is not created by altool - it's created by stapler (or as a result of running stapler). I found it out after checking the app contents after running each stage.

               

              The volume where all the work files are located is APFS (as well as my startup volume). The final dmg is HFS+ - albeit it's irrelevant as the Apple Double file is created before copying to dmg.

               

              Here's the file hex ump if it makes any difference:

               

              00000000  00 05 16 07 00 02 00 00  4d 61 63 20 4f 53 20 58  |........Mac OS X|

              00000010  20 20 20 20 20 20 20 20  00 02 00 00 00 09 00 00  |        ........|

              00000020  00 32 00 00 00 7e 00 00  00 02 00 00 00 b0 00 00  |.2...~..........|

              00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

              *

              00000050  00 00 00 00 41 54 54 52  00 00 00 00 00 00 00 b0  |....ATTR........|

              00000060  00 00 00 a0 00 00 00 10  00 00 00 00 00 00 00 00  |................|

              00000070  00 00 00 00 00 00 00 01  00 00 00 a0 00 00 00 10  |................|

              00000080

               

              Opening it in TextEdit also reveals this string:

              com.apple.lastuseddate#PS

               

              Like I mentioned, this ._Info.plist file is only created in one app so far. When I notarize another somewhat similar app this doesn't happen. I have several other apps I didn't check yet.

               

               

              So to summarize the situation:

               

              -The app is successfully signed, notarized and stapled on APFS volume.

               

              -And yet, the stapler's action results in addition of an invisible file that invalidates the signature.

               

              -The app is then copied to an custom installer on dmg.

               

              -The installer is successfully notarized with altool

               

              -BUT, at the last stage, on attempt to staple the installer I get Error 65.

               

               

              SOLUTION:

               

              Delete the ._Info.plist file from the app prior to copying it to the installer.

               

               

              I, frankly, can't comprehend how a successfully notarized app can end up being invalid for further processing due to the actions of the very notarization tools. And that developers should hunt for and delete some invisible file from the app to make it valid again.

               

               

              Thanks,

              Leo

                • Re: altool injects malicious invisible files into notarized apps
                  eskimo Apple Staff Apple Staff (12,975 points)

                  Thanks for all the extra info.  You wrote:

                  Here's the file hex [dump] if it makes any difference:

                  Yeah, that’s what I was hoping to see.  This definitely looks like an Apple Double header.  Pulling it apart according to the format documented in RFC 1740 I see this:

                  Header
                  ------
                  magicNum    uint32_t    0x00051607
                  versionNum  uint32_t    0x00020000
                  filler      16 bytes    `Mac OS X        `
                  numEntries  uint16_t    2
                  
                  Entry 0
                  -------
                  entryID     uint32_t    9 == AS_FINDERINFO
                  entryOffset uint32_t    0x00000032
                  entryLength uint32_t    0x0000007E
                  
                  Entry 1
                  -------
                  entryID     uint32_t    2 == AS_RESOURCE
                  entryOffset uint32_t    000000B0
                  entryLength uint32_t    00000000

                  There are some oddities here though.  Specifically, the length of the AS_FINDERINFO entry causes it to extend beyond the end of the file.  Did you dump the whole file?

                  Regardless, I’m pretty darned sure that stapler didn’t put that there.  stapler is a very UNIX-y tool; it doesn’t mess with Apple-specific metadata at all.

                  Can you describe your notarisation process in more detail?  What type of file are you submitting to the notary service?  A zip archive?

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: altool injects malicious invisible files into notarized apps
                      Leo Braun Level 1 Level 1 (0 points)

                      Ok, I isolated some additional stages - and here's another update:

                       

                      You're right, it's not stapler that generates the ._Info.plist file.

                       

                      The file is created while unzipping the app after running altool and before running stapler.

                       

                      The unzipping is done by running the unzip binary via NSTask. I looked again through the unzip options - but couldn't find anything that would prevent it from generating the file. The ._Info.plist file is not created when unzipping manually in the Finder.

                       

                      Also, I included the full hexdump of the file at the bottom (although it's now probably irrelevant considering the above).

                       

                       

                      This new info, however, doesn't change the nature of the original issue.

                       

                      Let me lay it down again, more accurately:

                       

                      -The app is successfully signed, zipped, and notarized by altool

                       

                      -The app is then unzipped in order to be stapled - now with the ._Info.plist file inside.

                       

                      -The app is successfully stapled.

                       

                      -The app is then copied to a custom installer on dmg.

                       

                      -The installer is successfully notarized with altool

                       

                      -BUT, at the last stage, on attempt to staple the installer I get Error 65 (due to the ._Info.plist file inside the app).

                       

                       

                      I still don't understand how is it possible for the app to be successfully notarized - while at the same time its signature is in fact compromised. And that developers should hunt for and delete some invisible file from the app to make it valid again.

                       

                      In my view, IF this invisible file presents a problem, then stapler should give an error while stapling the original app - which error would describe exactly what the issue is and advise on a possible solution.

                       

                      Alternatively, IF the app is successfully notarized and stapled - then it should mean that there's no problem with the app and the app won't cause any issues during further processing (such as notarizing the installer).

                       

                       

                      Hope it all makes sense by now!

                       

                       

                      Thanks,

                      Leo

                       

                       

                      00000000  00 05 16 07 00 02 00 00  4d 61 63 20 4f 53 20 58  |........Mac OS X|

                      00000010  20 20 20 20 20 20 20 20  00 02 00 00 00 09 00 00  |        ........|

                      00000020  00 32 00 00 00 7e 00 00  00 02 00 00 00 b0 00 00  |.2...~..........|

                      00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

                      *

                      00000050  00 00 00 00 41 54 54 52  00 00 00 00 00 00 00 b0  |....ATTR........|

                      00000060  00 00 00 a0 00 00 00 10  00 00 00 00 00 00 00 00  |................|

                      00000070  00 00 00 00 00 00 00 01  00 00 00 a0 00 00 00 10  |................|

                      00000080

                      Leos-iMac:~ startup$ hexdump -C /Volumes/iMac/Users/me/Development/Apple/Notary/Staging/Output\ Factory.app/Contents/._Info.plist

                      00000000  00 05 16 07 00 02 00 00  4d 61 63 20 4f 53 20 58  |........Mac OS X|

                      00000010  20 20 20 20 20 20 20 20  00 02 00 00 00 09 00 00  |        ........|

                      00000020  00 32 00 00 00 7e 00 00  00 02 00 00 00 b0 00 00  |.2...~..........|

                      00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|

                      *

                      00000050  00 00 00 00 41 54 54 52  00 00 00 00 00 00 00 b0  |....ATTR........|

                      00000060  00 00 00 a0 00 00 00 10  00 00 00 00 00 00 00 00  |................|

                      00000070  00 00 00 00 00 00 00 01  00 00 00 a0 00 00 00 10  |................|

                      00000080  00 00 1a 63 6f 6d 2e 61  70 70 6c 65 2e 6c 61 73  |...com.apple.las|

                      00000090  74 75 73 65 64 64 61 74  65 23 50 53 00 00 00 00  |tuseddate#PS....|

                      000000a0  55 1c 35 5d 00 00 00 00  b8 e0 0c 38 00 00 00 00  |U.5].......8....|

                      000000b0

                        • Re: altool injects malicious invisible files into notarized apps
                          eskimo Apple Staff Apple Staff (12,975 points)

                          The unzipping is done by running the unzip binary via NSTask.

                          OK.  What’s going here is:

                          1. You have some sort of Mac-specific metadata on your Info.plist.

                          2. You are zipping it with a tool (ditto, right?) that understands Mac-specific metadata, and thus has added an AppleDouble file to the zip archive.

                          3. You are unzipping it with a tool that does not understand Mac-speciifc metadata (unzip) and that’s extracted the AppleDouble file as a normal file.

                          There are two things you need to fix:

                          • Remove the Mac-specific metadata from your file before you start this whole process.  This metadata is not needed in your case [1] and, depending on your specific distribution channel, it can cause problems.

                          • Do you zip and unzip with the same tool.  Doing the zip with one tool and the unzip with another is likely to cause ongoing problems.


                          Now, to address your specific points:

                          I still don't understand how is it possible for the app to be successfully notarized - while at the same time its signature is in fact compromised.

                          Because the file you submitted was correctly signed; you broke the signature when you unzipped it using unzip.

                          In my view, IF this invisible file presents a problem, then staplershould give an error while stapling the original app - which error would describe exactly what the issue is and advise on a possible solution.

                          stapler does not currently check the code signature of the thing it’s stapling too.  Rather, it reads some basic info from the code signature (the cdhash), uses that to look up the ticket, and then applies that ticket.

                          If you’d like stapler to be smarter about this, feel free to file a bug report with your suggestions.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                          [1] In some very rare cases Mac-specific metadata is required.  In your case, however, it looks like the metadata in question is com.apple.lastuseddate, which your customers don’t need to know.

                  • Re: altool injects malicious invisible files into notarized apps
                    john daniel Level 4 Level 4 (570 points)

                    It sounds like you are using the command-line zip or a 3rd party zip tool. Use the "Compress" option from the Finder's context menu.

                      • Re: altool injects malicious invisible files into notarized apps
                        Leo Braun Level 1 Level 1 (0 points)

                        John,

                         

                        Thanks for your answer! I understand where you're coming from.

                         

                        But:

                         

                        -First, like I mentioned in the last reply to eskimo, the invisible file is created at the stapling stage - not after being zipped.

                         

                        -Second, I have several apps and my notarization process is fully automated - so I'm of course using a command line tool (ditto in this case). These are all legitimate tools - and fortunately there are no reasons to resort to manual operations in the Finder!

                        • Re: altool injects malicious invisible files into notarized apps
                          Leo Braun Level 1 Level 1 (0 points)

                          Hi John,

                           

                          I investigated further - and here's an update:

                           

                          I found out that technically you're indeed right. The ._Info.plist file is in fact created when unzipping the app using the unzip binary (via NSTask) - prior to the stapling stage. I elaborate on this more in the new reply to eskimo.

                           

                          The invisible file is indeed not created when unzipping manually in the Finder.

                           

                          This, of course, still doesn't mean that we should do things manually - because it's simply out of the question.

                           

                          But I believe that the mere fact that we have this discussion is a part of a bigger issue: that the entire notarization process is just not well designed and implemented. Because if it was well designed and implemented, we wouldn't have to go that deep into this rabbit hole and split hair about what zipping tool is more suitable in order avoid some obscure invisible files in the year 2020.

                           

                           

                          Leo

                            • Re: altool injects malicious invisible files into notarized apps
                              john daniel Level 4 Level 4 (570 points)

                              I found out that technically you're indeed right.

                               

                              What's the point of the "technically" qualifier?

                               

                              The ._Info.plist file is in fact created when unzipping the app using the unzipbinary (via NSTask) - prior to the stapling stage.

                               

                              So eliminate that step. Problem solved.

                               

                              This, of course, still doesn't mean that we should do things manually - because it's simply out of the question.

                               

                              What is "things"? I've never understood this desire to automate notarization. Technically , it is possible. However, the only reason for doing this is automating the building and distribution of software over the internet. That is a bad idea and completely unnecessary. Clearly, it wasn't something that Apple had envisioned. It is like the Tragedy of the Commons. If everyone starts doing this, then bad things could happen for everyone. Either it slows down notarization for everyone, Apple starts throttling, or maybe Apple comes up with something even more clever than that.

                               

                              the entire notarization process is just not well designed and implemented. Because if it was well designed and implemented, we wouldn't have to go that deep into this rabbit hole and split hair about what zipping tool is more suitable in order avoid some obscure invisible files in the year 2020.

                               

                              That's a soapbox you probably shouldn't get up on. Notarization is drop-dead, fall-off-a-log easy. It really is. I have yet to see a single one of these notarization threads that wasn't caused by a developer doing any number tasks that range from merely unnecessary to just outlandish. Of course, edge cases are always possible. But if someone wants to get up on a notarization soapbox, they should realize that the evidentary burden of proof is very high.

                               

                              Yes, it is 2020. But the nature of zip files on the Macintosh hasn't changed much since day one. The Mac pre-dated PkZip, after all. It was designed on Windows and completely unsuitable for the Mac. The world being what it is, zip files are now standard practice. But that doesn't fundamentally change anything about them. If you are going to use zip files on the Mac, you have to do it properly. It is unfortunate that you would go to all this effort to automate a Mac software build and distribution system without understanding this basic issue. What else are you missing?

                               

                              I'll cut to the chase. If you want to automate unecessary tasks like unzipping and clog up the notariztion pipeline, go for it. Use ditto instead. However, my suggestion would be to review Apple's published instructions for customizing the notarization workflow. What does it say about unzipping?

                                • Re: altool injects malicious invisible files into notarized apps
                                  Leo Braun Level 1 Level 1 (0 points)

                                  Thanks for the detailed answer, John.

                                   

                                   

                                  > So eliminate that step. Problem solved.

                                   

                                  I don't see how I can eliminate unzipping. If there is a way, I'll be glad to learn about it.

                                   

                                   

                                  Anyway, I respectfully disagree with your views on automation:

                                   

                                   

                                  > I've never understood this desire to automate notarization.

                                   

                                   

                                  Manual notarization might be acceptable if you have a single app which you update once a month, if even.

                                   

                                  Like I mentioned, I have multiple apps that I update often.

                                   

                                  Manual notarization involves many repetitive, tedious tasks and is a huge waste of time and resources. Repetitive, tedious tasks are:

                                   

                                  -counter-productive, and

                                  -degrading the quality of human life.

                                   

                                  It's never acceptable for people who respect their time - be it notarization or otherwise.

                                   

                                  Fortunately, Apple understands this and does provide developers with tools to automate the process. For which I can only be grateful - even though it doesn't fix the major principal flaws of the notarization process.

                                   

                                   

                                  > I have yet to see a single one of these notarization threads that wasn't caused by a developer doing any number tasks that

                                  > range from merely unnecessary to just outlandish.

                                   

                                   

                                  Exactly - and that's a part of the bigger issue I'm talking about.

                                   

                                  It's like "you're holding it wrong" issue with iPhone 4. Obviously, the way you're holding a phone shouldn't make a difference in the first place.

                                   

                                  Similarly, if notarization was implemented the way it should, you wouldn't even see those threads. Because developers wouldn't even have a chance to run into these issues.

                                   

                                   

                                  > Yes, it is 2020. But the nature of zip files on the Macintosh hasn't changed much since day one. (etc.)

                                   

                                   

                                  Once again - exactly what I'm talking about.

                                   

                                  Your excursion in the history of zipping is part of the problem. Because the issue of zipping shouldn't even come up on the level of developer implementation.

                                   

                                  If notarization process requires something to be zipped/unzipped, then Apple tools should do it automatically under the hood - without having to involve developers in the details of this process.

                                   

                                   

                                  > ...my suggestion would be to review Apple's published instructions for customizing the notarization workflow.

                                  > What does it say about unzipping?

                                   

                                   

                                  I guess there's a caveat in this question but - you tell me. I don't see anything. Needless to say, I find Apple's notarization documentation inadequate and detached from the way developers work in real life. But that's a different topic.

                                   

                                   

                                  Don't get me wrong - I'm just as Apple fan as anyone else here, using Macs since 1994 and developing since early 00s. But when it comes to notarization, Apple isn't at its best so far. Apple can - and should - do better.

                                   

                                   

                                  Leo

                                    • Re: altool injects malicious invisible files into notarized apps
                                      john daniel Level 4 Level 4 (570 points)

                                      If notarization process requires something to be zipped/unzipped, then Apple tools should do it automatically under the hood - without having to involve developers in the details of this process.

                                       

                                      Please use the command-Line tool "man" in the Terminal.app to review usage of the "ditto" tool.


                                      I guess there's a caveat in this question but - you tell me. I don't see anything. Needless to say, I find Apple's notarization documentation inadequate and detached from the way developers work in real life.


                                      There is no unzipping step. When you notarize a zip file, you are notarizing the app it contains. It is not practical to upload an app bundle which is just a complex directory tree. You are only zipping for uploading. After uploading, you delete the zip. Then you staple the app. Then you build a new zip file using "ditto", as described in the documentation.