How do I support VoiceOver in a SpriteKit game?

I'm working on a card game in SpriteKit, and I think it's a candidate for VoiceOver support.


I'm having a hard time finding resources on how to properly do this, however. It seems like VoiceOver is mainly designed to play nice with UIKit controls, but not so much SpriteKit scenes. Near as I can tell, I need to treat my entire SpriteKit view as a VoiceOver "container" view and then use the informal protocol of having accessibilityElementCount(), etc., report the accessible elements.


I have this working, to an extent. I can make my placards readable and my buttons clickable with VoiceOver. But I'm running into a lot of other daunting issues, and it just seems like I must be missing something. For instance:


* VoiceOver auto-selects a node I don't want it to when it starts up and starts reading it. This is a problem because it's not the "right" node I'd want selected, and I often want to have it say something to the user when a new scene is loaded, and auto-selecting the node cancels that. How do I prevent this behavior?


* Sometimes, VoiceOver selects an SKLabelNode and speaks its text, which is weird because I didn't code it to do that. It implies that there might be some kind of intentional VoiceOver support behind the scenes. Is there? Am I missing some documentation somewhere that describes this? How do I turn it off if I want to handle speaking that text at a higher level (i.e., if I have two SKLabelNodes showing a sentence over two lines, I want one rollover area that speaks the whole sentence, not two rollover areas that speak the first half and the second half of a sentence)?


* I have a reference scene where I have a bunch of cards in a scrollable area, where you can scroll through them, tap on one, and get information about it. How do I translate that sort of interaction to something that works with VoiceOver? Normally, this would be done with some kind of paged control (I think?), but there's no equivalent in SpriteKit that I'm aware of. Am I supposed to roll my own solution? Do I need to translate that to something that only uses buttons (which feels like it would be an inferior solution for people who aren't relying on VoiceOver). What are the best practices here?


* How do I just let the player know something happened to something on the screen that isn't a UI layout change? - i.e., "You drew an ace of spades and a two of clubs" kind of information. The cards all draw to the same place on the screen, so it's not really a layout change, but it's different information on the screen. (Or is it a layout change because there are cards in those "wells" where there were none before?)


* And if I do change the layout, how do I tell the VoiceOver system that the layout has changed without it deciding to change the current selection or otherwise taking over? Again, if a new card comes out, it should be tappable, but I'd rather VoiceOver speak something to this effect rather than trigger the default thing where it goes and tries to auto-select something.


Clearly, I'm adrift a bit here. I'm not sure I'm expressing myself well here because I'm still a little fuzzy on how it should all work, and the documentation I'm reading isn't really clearing it up for me.


What I'd really like to see is an in-depth discussion about how to properly add VoiceOver support to a SpriteKit game that hits these sorts of topics and goes through the proper structure for supporting VoiceOver, showing examples of how it's supposed to be done. Anyone have any resources they can point me to?

Replies

Okay, after lots of digging and experimentation, I have some results I can share, in case anyone else encounters these issues.



The basic model for supporting VoiceOver in SpriteKit turns out to be very simple. SKNodes conform to the same protocols that most UIKit elements do. For any given node, you can set isAccessibleElement to true, set the accessibility label, accessibility hint, and accessibility traits, and VoiceOver will treat them accordingly with no other effort from you. In particular, there's no need to implement any of the informal accessibility protocols described for custom UIView subclasses (although you can if you need that level of control, such as to control the swipe order of elements).



HOWEVER, there is currently an issue which interferes with this system in some cases.



VoiceOver touches are propagated down through your view hierarchy and cancelled if at any point in the path down to your node the touch is outside the LOCAL bounds of the node. This means that the touchable area for any node is the intersection of its own bounding box and the bounding boxes of all of its parents. If this intersection is null, then the item is inaccessible, and any touch registers on the SKView instead.



One of the consequences of this is that anything that is a child of an SKNode will become inaccessible with VoiceOver, because an SKNode has no width or height, and thus, it will block all discoverability of any of its children because the intersection of its bounding box with any other node will always be empty. If you use SKNodes to organize your scene, they will kill all accessibility for any of their descendants.



This can cause all sorts of problems. For instance, if you are implementing the informal accessibility "container" protocols, this can cause bad things to happen, like breaking out of the responder chain and causing swipes to the next item to go to the status bar (carrier bar strength, time, etc.) instead of to what you would expect.



Because of the above, currently the easiest way to support VoiceOver is to ensure that all of your accessible elements are simply direct children of the scene. If you cannot have your elements be direct children of the SKScene, you can work around this issue by parenting any accessible element you want to be available to a node that encompasses it in size, such as a transparent SKSpriteNode instead of an SKNode. But depending on what you're doing, this can cause its own problems. In general, you should try to keep your hierarchies as "flat" as possible.



Ideally, VoiceOver would consider the accumulated frame of a node and all its children before deciding to reject a touch because of the above issues. I have filed a radar with this issue: #28355567.

I can't thank you enough. The writing, thoroughness, style, structure, commentary... it's too good!


THANK YOU!!!


I hope this helps others as much as it's about to help me.


And thank you for the radar filing, too. Amazing. You're champion!