Accessibility API of the Books application

UI Frameworks AppKit
O.W.Grant OP
Created Jun ’20
Replies 5
Boosts 0
Views 2.0k
Participants 3

Hello

For my word processing app I need to be able to retrieve a specific paragraph of a book that is opened in Books.app on Mac. Using Accessibility APIs I am able to descend the tree of views down to AXStaticText, however whenever I try to read AXValue I always get nil. Also, I cannot read that text using Apple Script (of course, Script Editor is enabled as an Accessibility app in Security & Privacy)



Xcode's Accessibility Inspector, however, returns correct text of the paragraph in Books I am pointing it to.



I was wondering if it could be due to some DRM protection? But if so, I don't understand, how come Accessibility Inspector is able to read the text?

Thanks!

Replies  5
Boosts  0
Views  2.0k
Participants  3
cheeseb OP
Jun ’20

Apple's Books application is known to contain some "mismatch" errors in its implementation of accessibililty. That means it portrays the UI element hierarchy differently in one direction (for example, in a screen reader that starts with a leaf element on the screen) than in the other direction (for example, in a browser that starts with the application UI element. Try the free trial of my UI Browser application to explore this possibility. https://pfiddlesoft.com/uibrowser

0
O.W.Grant OP
Jun ’20

I also tried 3rd party UI Browser app. It doesn't show the labels' text either, only nil's and "" instead of the actual values.

The only app that is able to read is Apple's Accessibility Inspector.


I checked the entitlements of the Accessibility Inspector, and it contains these values:

<dict>

<key>com.apple.private.accessibility.inspection</key>

<true/>

<key>com.apple.private.speech.synthesis.custom.voices.allow</key>

<true/>

<key>com.apple.private.tcc.allow</key>

<array>

<string>kTCCServiceAccessibility</string>

</array>

</dict>


I understand that com.apple.private.tcc.allow basically bypasses the need to add the app to Security & Privacy. Could it be that "com.apple.private.accessibility.inspection" is the one that allows the inspector to access text fields in the Books app? Sadly these entitlements cant be added to non-apple apps, though...

0
KMT OP
Jun ’20

>I was wondering if it could be due to some DRM protection?


I believe so, certainly.


> But if so, I don't understand, how come Accessibility Inspector is able to read the text?


Devs shouldn't be surprised that the OS can access data they can't.

0
O.W.Grant OP
Jun ’20

It doesn't work with Books in context of reading an element's value.

Have you tried it?

0
cheeseb OP
Jun ’20

Yes, I have explored Apple's Books application in depth with UI Browser. If you give it a try as I suggested, use UI Browser's browser view to traverse the Books UI element hierarchy down through several levels of "AXGroup" elements until you come to the "AXWebArea" element. Then look in UI Browser's Attributes drawer to see the details of how Books uses Apple's specialized AXWebArea accessibility API to parse text. AXWebArea is a very complex API, and it is not easy to use with GUI Scripting or code. UI Browser can track the attributes and help you write GUI Scripting scripts that will retrieve the text values, but UI Browser is not designed to retrieve AXWebArea text values directly. Apple's Accessibility Inspector utility does retrieve the text values directly.

0
Accessibility API of the Books application
First post date Last post date
 
 
Q