INPlayMediaIntent `mediaSearch` mediaName unreliable when searching for playlists

We are working with an app that uses the INPlayMediaIntent to allow users to select and play music using Siri.

In building out this feature, we have noticed that when selecting playlists to play, Siri will consistently leave out information from the intent that we are use to resolve the media to play in the app.

It seems that there is generally no rhyme or reason as to why some information is left out.

Walking through a couple test cases, here is the phrase and corresponding mediaSearch that we receive when testing:

  • "Hey Siri, play the playlist happy songs in the app <app name>" (this is a working example)
▿ Optional<INMediaSearch>
  - some : <INMediaSearch: 0x114050780> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = happy songs;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}
  • "Hey Siri, play the playlist my favorites in the app <app name>" (this fails with a null mediaName)
▿ Optional<INMediaSearch>
  - some : <INMediaSearch: 0x114050600> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = <null>;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}
  • "Hey Siri, play the playlist working out playlist in the app <app name>" (this fails as the term "playlist" is excluded)
▿ Optional<INMediaSearch>
  - some : <INMediaSearch: 0x114050ae0> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = working out;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}
  • "Hey Siri, play the playlist recently added in the app <app name>" (this fails with a null mediaName)
▿ Optional<INMediaSearch>
  - some : <INMediaSearch: 0x1140507e0> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = <null>;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}

Based on the above, Siri seems to ignore playlists named "Recently Added", "My Favorites", and playlists that have the word "playlist" in them such as "Working Out Playlist".

To rectify this, we attempted to set the INVocabulary for the playlist titles that a user has in the app, as suggested in this WWDC session: https://developer.apple.com/videos/play/wwdc2020/10060/

let vocabulary = INVocabulary.shared()
vocabulary.setVocabularyStrings(NSOrderedSet(array: [
    "my favorites",
    "recently added",
    "working out playlist"
]), of: .mediaPlaylistTitle);

This seems to have no effect. We understand the note in https://developer.apple.com/documentation/sirikit/registering_custom_vocabulary_with_sirikit/ stating that "a few minutes" should be waited before testing custom vocabulary, but waiting upwards of 20 minutes and even restarting the device did not result in any of the custom vocabulary making a difference.

If these playlist names are set in AppIntentVocabulary.plist, "Recently Added" and "My Favorites" are able to be discovered as playlists, but the other failed test cases remain failing. The obvious shortcoming here is that these are not dynamic.

<key>ParameterVocabularies</key>
<array>
    <dict>
        <key>ParameterNames</key>
        <array>
            <string>INPlayMediaIntent.playlistTitle</string>
        </array>
        <key>ParameterVocabulary</key>
        <array>
            <dict>
                <key>VocabularyItemIdentifier</key>
                <string>working out playlist</string>
                <key>VocabularyItemSynonyms</key>
                <array>
                    <dict>
                        <key>VocabularyItemPhrase</key>
                        <string>working out playlist</string>
                    </dict>
                </array>
            </dict>
            <dict>
                <key>VocabularyItemIdentifier</key>
                <string>recently added</string>
                <key>VocabularyItemSynonyms</key>
                <array>
                    <dict>
                        <key>VocabularyItemPhrase</key>
                        <string>recently added</string>
                    </dict>
                </array>
            </dict>
            <dict>
                <key>VocabularyItemIdentifier</key>
                <string>my favorites</string>
                <key>VocabularyItemSynonyms</key>
                <array>
                    <dict>
                        <key>VocabularyItemPhrase</key>
                        <string>my favourites</string>
                    </dict>
                    <dict>
                        <key>VocabularyItemPhrase</key>
                        <string>my favorites</string>
                    </dict>
                </array>
            </dict>
        </array>
    </dict>
</array>

Given the above, our questions are as follows:

  1. Is there documentation surrounding how Siri may pass along the mediaSearch in INPlayMediaIntent and how/why information may be left out?

  2. Why does setting custom vocabulary with INVocabulary seem to have no effect, yet the same vocabulary in AppIntentVocabulary does have an effect?

  3. Is the functionality we are experiencing to be expected, or should this be reported as a bug?

We've published the test app that we are using for debugging this functionality at this link: https://github.com/awojnowski/SiriTest

After looking into this further, is it possible that vocabulary registered with INVocabulary will appear as <null> instead of the vocabulary when on development devices?

For example, on our main development device, we have a playlist registered via INVocabulary named "Country Bangers". When using the phrase "Hey Siri, play the playlist Country Bangers in <app name>", we receive the following mediaSearch:

<INMediaSearch: 0x101c587e0> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = <null>;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}

However, if we use the same phrase on a different test device without that playlist name in its registered vocabulary, we receive a mediaSearch that is, in fact, populated with the playlist name:

<INMediaSearch: 0x126054780> {
    reference = 0;
    mediaType = 5;
    sortOrder = 0;
    albumName = <null>;
    mediaName = country bangers;
    genreNames = (
    );
    artistName = <null>;
    moodNames = (
    );
    releaseDate = <null>;
    mediaIdentifier = <null>;
}

Our engineering teams need to investigate this issue, as resolution may involve changes to Apple's software. I'd greatly appreciate it if you could open a bug report, include your intents and link to this post, and reply here with the FB number once you do. Bug Reporting: How and Why? has tips on creating your bug report.

Filed under FB14872048.

INPlayMediaIntent `mediaSearch` mediaName unreliable when searching for playlists
 
 
Q