8 Replies
      Latest reply on Jan 16, 2020 5:17 AM by simran21
      simran21 Level 1 Level 1 (0 points)

        Hello,

         

        We have file system filter. Recently on Catalina macOS we found that following tccd related files are stuck at open call:

         

        • /private/var/folders/tb/jjk74dn51p5_vgz844dpvfbw0000gn/T/com.apple.tccd/TemporaryItems/(A Document Being Saved By tccd)/keys
        • /Library/Application Support/com.apple.TCC/AdhocSignatureCache/keys

         

        At the same time, tccd process in Activity monitor also stuck at _NSWriteDataToFileWithExtendedAttributes / _NSReadBytesFromFileWithExtendedAttributes. Please check below samples of tccd process : -

         

        Thread_165622   DispatchQueue_24: com.apple.tcc.AdhocSignatureCache  (serial)
            + 2852 start_wqthread  (in libsystem_pthread.dylib) + 15  [0x7fff7224857b]
            +   2852 _pthread_wqthread  (in libsystem_pthread.dylib) + 290  [0x7fff7224871b]
            +     2852 _dispatch_workloop_worker_thread  (in libdispatch.dylib) + 598  [0x7fff71ffaa9e]
            +       2852 _dispatch_lane_invoke  (in libdispatch.dylib) + 363  [0x7fff71ff1452]
            +         2852 _dispatch_lane_serial_drain  (in libdispatch.dylib) + 597  [0x7fff71ff0ace]
            +           2852 _dispatch_client_callout  (in libdispatch.dylib) + 8  [0x7fff71feb50e]
            +             2852 _dispatch_call_block_and_release  (in libdispatch.dylib) + 12  [0x7fff71fea583]
            +               2852 __53-[TCCDAdhocSignatureCache getSignatureForStaticCode:]_block_invoke.114  (in tccd) + 211  [0x10c50462e]
            +                 2852 -[TCCDAdhocSignatureCache saveKeysToDirectory]  (in tccd) + 189  [0x10c502fc6]
            +                   2852 _NSWriteDataToFileWithExtendedAttributes  (in Foundation) + 1005  [0x7fff3d1d2d41]
            +                     2852 close  (in libsystem_kernel.dylib) + 10  [0x7fff721851aa]
        

         

        OR

         

         2641 Thread_5755   DispatchQueue_27: com.apple.tcc.AdhocSignatureCache  (serial)
            + 2641 start_wqthread  (in libsystem_pthread.dylib) + 15  [0x7fff688a957b]
            +   2641 _pthread_wqthread  (in libsystem_pthread.dylib) + 290  [0x7fff688a971b]
            +     2641 _dispatch_workloop_worker_thread  (in libdispatch.dylib) + 598  [0x7fff6865ba9e]
            +       2641 _dispatch_mach_invoke  (in libdispatch.dylib) + 481  [0x7fff6866263e]
            +         2641 _dispatch_lane_serial_drain  (in libdispatch.dylib) + 263  [0x7fff68651980]
            +           2641 _dispatch_mach_msg_invoke  (in libdispatch.dylib) + 435  [0x7fff68661aeb]
            +             2641 _dispatch_client_callout4  (in libdispatch.dylib) + 9  [0x7fff6864c5ae]
            +               2641 _xpc_connection_mach_event  (in libxpc.dylib) + 927  [0x7fff688e9158]
            +                 2641 _xpc_connection_call_event_handler  (in libxpc.dylib) + 56  [0x7fff688eaf68]
            +                   2641 __main_block_invoke.167  (in tccd) + 55  [0x10e76fcd5]
            +                     2641 handle  (in tccd) + 3308  [0x10e770a0d]
            +                       2641 do_TCCAccessRequest  (in tccd) + 8508  [0x10e77642c]
            +                         2641 -[TCCDAccessIdentity matchesCodeRequirementData:]  (in tccd) + 345  [0x10e787ecb]
            +                           2641 -[TCCDPlatformMacOS adhocSignStaticCode:]  (in tccd) + 169  [0x10e7a45b3]
            +                             2641 -[TCCDAdhocSignatureCache getSignatureForStaticCode:]  (in tccd) + 199  [0x10e79422d]
            +                               2641 _dispatch_lane_barrier_sync_invoke_and_complete  (in libdispatch.dylib) + 60  [0x7fff68658567]
            +                                 2641 _dispatch_client_callout  (in libdispatch.dylib) + 8  [0x7fff6864c50e]
            +                                   2641 __53-[TCCDAdhocSignatureCache getSignatureForStaticCode:]_block_invoke  (in tccd) + 312  [0x10e7943d6]
            +                                     2641 -[TCCDAdhocSignatureCache loadSignatureWithUUID:]  (in tccd) + 124  [0x10e7930db]
            +                                       2641 +[NSData(NSData) dataWithContentsOfURL:options:error:]  (in Foundation) + 61  [0x7fff3384473a]
            +                                         2641 -[NSData(NSData) initWithContentsOfFile:options:maxLength:error:]  (in Foundation) + 111  [0x7fff3383e075]
            +                                           2641 _NSReadBytesFromFileWithExtendedAttributes  (in Foundation) + 160  [0x7fff338265d6]
            +                                             2641 __open  (in libsystem_kernel.dylib) + 10  [0x7fff687e6192]
            2641 Thread_5996   DispatchQueue_11: com.apple.root.default-qos.overcommit  (concurrent)
            + 2641 start_wqthread  (in libsystem_pthread.dylib) + 15  [0x7fff688a957b]
            +   2641 _pthread_wqthread  (in libsystem_pthread.dylib) + 290  [0x7fff688a971b]
            +     2641 _dispatch_workloop_worker_thread  (in libdispatch.dylib) + 598  [0x7fff6865ba9e]
            +       2641 _dispatch_mach_invoke  (in libdispatch.dylib) + 481  [0x7fff6866263e]
            +         2641 _dispatch_lane_serial_drain  (in libdispatch.dylib) + 263  [0x7fff68651980]
            +           2641 _dispatch_mach_msg_invoke  (in libdispatch.dylib) + 435  [0x7fff68661aeb]
            +             2641 _dispatch_client_callout4  (in libdispatch.dylib) + 9  [0x7fff6864c5ae]
            +               2641 _xpc_connection_mach_event  (in libxpc.dylib) + 927  [0x7fff688e9158]
            +                 2641 _xpc_connection_call_event_handler  (in libxpc.dylib) + 56  [0x7fff688eaf68]
            +                   2641 __main_block_invoke.167  (in tccd) + 55  [0x10e76fcd5]
            +                     2641 handle  (in tccd) + 3308  [0x10e770a0d]
            +                       2641 do_TCCAccessRequest  (in tccd) + 8508  [0x10e77642c]
            +                         2641 -[TCCDAccessIdentity matchesCodeRequirementData:]  (in tccd) + 345  [0x10e787ecb]
            +                           2641 -[TCCDPlatformMacOS adhocSignStaticCode:]  (in tccd) + 169  [0x10e7a45b3]
            +                             2641 -[TCCDAdhocSignatureCache getSignatureForStaticCode:]  (in tccd) + 199  [0x10e79422d]
            +                               2641 _dispatch_sync_f_slow  (in libdispatch.dylib) + 171  [0x7fff6865840e]
            +                                 2641 __DISPATCH_WAIT_FOR_QUEUE__  (in libdispatch.dylib) + 270  [0x7fff686587ab]
            +                                   2641 _dispatch_event_loop_wait_for_ownership  (in libdispatch.dylib) + 498  [0x7fff686682fe]
            +                                     2641 _dispatch_kq_poll  (in libdispatch.dylib) + 247  [0x7fff68667844]
            +                                       2641 kevent_id  (in libsystem_kernel.dylib) + 10  [0x7fff687e6c22]
        

         

        Which process have file handle :  tccd or our process trying to open a file?

        Please  suggest document  for tccd to analyze the issue.

         

        Thanks ,

        Simran

        • Re: tccd process stuck with file system filter
          eskimo Apple Staff Apple Staff (12,705 points)

          We have file system filter.

          What do you mean by “file system filter”?

          Share and Enjoy

          Quinn “The Eskimo!”
          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
          let myEmail = "eskimo" + "1" + "@apple.com"

            • Re: tccd process stuck with file system filter
              simran21 Level 1 Level 1 (0 points)

              Its Kauth kext which monitors file activities.

               

              Thanks,

              Simran

                • Re: tccd process stuck with file system filter
                  eskimo Apple Staff Apple Staff (12,705 points)

                  Deadlock is a common problem with kauth KEXTs.  Looking at a backtrace of the deadlocked process is only half the story.  What’s happening in your kauth KEXT at the time of the deadlock?

                  Share and Enjoy

                  Quinn “The Eskimo!”
                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                  let myEmail = "eskimo" + "1" + "@apple.com"

                    • Re: tccd process stuck with file system filter
                      simran21 Level 1 Level 1 (0 points)

                      Thanks for your response.

                       

                      Please check following details :

                       

                      1:

                      Kauth kext is basically providing events to user mode for scanning a file. It waits till scanning is done.

                      So kext is waiting for that thread (in user mode) which open a file , scan it and provide result to kauth. Here "open" call of tccd internal file (/Library/Application Support/com.apple.TCC/AdhocSignatureCache/keys)  is not completed  (it is stucked) and user mode thread is waiting for the same.

                       

                      2:

                      With "Signed and Notarized" build above issue did not occur. It is generated only with the build which is partially signed and not-notarized. (Here, only kexts are signed)

                       

                      Thanks,

                      Simran

                        • Re: tccd process stuck with file system filter
                          eskimo Apple Staff Apple Staff (12,705 points)

                          This situation is covered by the Deadlock Avoidance section of Technote 2127 Kernel Authorization.

                          Share and Enjoy

                          Quinn “The Eskimo!”
                          Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                          let myEmail = "eskimo" + "1" + "@apple.com"

                            • Re: tccd process stuck with file system filter
                              simran21 Level 1 Level 1 (0 points)

                              Hello,

                               

                              I have verified kauth for details. I have minimized location to generate events to folder  "/Library/Application Support/com.apple.TCC/AdhocSignatureCache" only.

                              after machine start , only 2 threads are waiting : which are from the same path with process tccd (root and user mode). no other threads are blocking/ waiting.

                               

                              Please check following points:

                              • In my daemon open call is blocking . what are the causes for this? why it is not returning ?
                              • How to identify why  open is stuck ?

                               

                              Thanks ,

                              Sheetal

                                • Re: tccd process stuck with file system filter
                                  eskimo Apple Staff Apple Staff (12,705 points)

                                  In my daemon open call is blocking.

                                  What does the in-kernel backtrace of the open call look like?

                                  Share and Enjoy

                                  Quinn “The Eskimo!”
                                  Apple Developer Relations, Developer Technical Support, Core OS/Hardware
                                  let myEmail = "eskimo" + "1" + "@apple.com"

                                    • Re: tccd process stuck with file system filter
                                      simran21 Level 1 Level 1 (0 points)

                                      Hello,

                                       

                                      While checking kernel details , I have seen following logs of sandbox and tccd daemons, please check below:

                                       

                                      1. Vnode action for "/Library/Application Support/com.apple.TCC/AdhocSignatureCache/F86FE4D8-7544-446E-B7B6-8C2440A00598" in kauth
                                      2. TCCD:
                                        1. Error reading signature from URL: url=/Library/Application Support/com.apple.TCC/AdhocSignatureCache/F86FE4D8-7544-446E-B7B6-8C2440A00598 error=Error Domain=NSCocoaErrorDomain Code=257 "The file “F86FE4D8-7544-446E-B7B6-8C2440A00598” couldn’t be opened because you don’t have permission to view it." UserInfo={NSFilePath=/Library/Application Support/com.apple.TCC/AdhocSignatureCache/F86FE4D8-7544-446E-B7B6-8C2440A00598, NSUnderlyingError=0x7fb8b0f22dd0 {Error Domain=NSPOSIXErrorDomain Code=13 "Permission denied"}}
                                      3. Sandbox: <Our user daemon> (230) System Policy: deny(1) file-read-data /Library/Application Support/com.apple.TCC/AdhocSignatureCache/F86FE4D8-7544-446E-B7B6-8C2440A00598

                                       

                                      Why sandbox denying to read tccd files ? and open call gets blocked in user mode?

                                       

                                      Thanks,

                                      Simran