Getting selected text from ImageAnalysisInteraction on UIImageView

Question

Created Jun ’22

Replies 5

Boosts 1

Views 2.2k

Participants 3

I work on an iOS app that displays images that often contain text, and I'm adding support for ImageAnalysisInteraction as described in this WWDC 2022 session. I have gotten as far as making the interaction show up and being able to select text and get the system selection menu, and even add my own action to the menu via the buildMenuWithBuilder API. But what I really want to do with my custom action is get the selected text and do a custom lookup-like thing to check the text against other content in my app.

So how do I get the selected text from an ImageAnalysisInteraction on a UIImageView? The docs show methods to check if there is selected text, but I want to know what the text is.

Answered by Mr. Jefferson in 761272022

There's an API for this in iOS 17 and later! ImageAnalysisInteraction.selectedText. https://developer.apple.com/documentation/visionkit/imageanalysisinteraction/4176689-selectedtext

Boost

Answer 1

Mr. Jefferson OP

Jun ’22

To clarify my use case: my app is an e-reader app focused on Biblical materials, with a cross-referencing system that can surface content from any downloaded books related to any verse or entity (people, places, things, ideas). There are lots of atlas-like resources with maps of the ancient near east in Biblical times, and it would be amazing to be able to select the name of a city on a map image and feed it into my existing functionality to find content related to that city name - verses, encyclopedia articles, etc.

[edit: this should have been a comment, but now I can't find a way to delete this "reply"]

0

Answer 2

Mr. Jefferson OP

Jul ’22

I've filed this as Feedback FB10794147.

0

Answer 3

josessolmaximo OP

Mar ’23

I am currently facing the same problem, enabled Live Text and couldn't get selected text so had to use VNRecognizeTextRequest instead. Have you found a solution to this?

0

Answer 4

Mr. Jefferson OP

Aug ’23

Accepted Answer

There's an API for this in iOS 17 and later! ImageAnalysisInteraction.selectedText. https://developer.apple.com/documentation/visionkit/imageanalysisinteraction/4176689-selectedtext

0

Answer 5

Jxiang OP

Aug ’23

How to change the system UIMenuController in VisionKit? I tried a few things but couldn't get it to work!

ex:


    func configMenuItems() {
        self.becomeFirstResponder()
        
        let menu       = UIMenuController.shared
        let paste      = UIMenuItem(title: "***".localized(), action: #selector(myPaste))
        let selectAll  = UIMenuItem(title: "zzz".localized(), action: #selector(mySelectAll))
        menu.menuItems = [paste, selectAll]
        menu.setMenuVisible(true, animated: true)
    }

0