Adding Subtitles From SEPARATE WebVTT File to AVAsset/PlayerItem

Media Technologies Audio
CuriosityStreamiOS OP
Created Apr ’16
Replies 13
Boosts 0
Views 11k
Participants 8

Hi All,


I apologive if this is a bit long. You can skip to part C to read the approach that I think could work (but has not worked for me yet).


We have m3u8 files hosted on a remote server. These files only point to video content; the master playlists do not include WebVTT content. The WebVTT file for each video is hosted remotely as well, but seperately.


We need a way to embed the subtitles into the video client-side in our iOS app (written in obj-c). After hacking away at this for a week, none of the following methods have worked (NOTE: each strategy was attempted seperately):


  • AVMediaSelectionGroup
    1. This was the first medthod I tried (as described in the WWDC '12 session What's New In HTTP Live Streaming). At 19:00 - or, if you have the slides, the slide titled "Enabling (Non-Forced) Subtitles" - the presenter takes an asset, gets the mediaSelectionGroupForMediaChacteristics and adds it to an AVPlayerItem. However, the asset is derived from a master playlist file that already has the WebVTT embedded, so this is not helpful in my case.
  • AVMutableComposition
    1. I created 2 AVURLAssets - one for my m3u8 file and one from my WebVTT file. I then created a AVMutableComposition, derived tracks from the 2 assets and inserted them into the composition. This did not work either. As described by an Apple staffer in this post: It is not possible to use AVMutableComposition to establish media selection groups for the subtitles, so you can't perform this sort of client side binding.
  • AVAssetReader/Writer + AVAssetResourceLoaderDelegate
    1. Apple has sample code for a command line tool called avsubtitleswriter. When run from the command line, the tool takes (1) an input path with the location of the movie file, (2) an output path to write to and (3) an array of paths pointing to subtitle files. Ultimatlely, it "creates a new movie file at the specified output location, with audio, video, and subtitles from the input source, adding subtitles from the provide subtitles file(s). Each subtitles file will become a subtitle track in the output movie". Using the sample code as a template, I performed the following steps:
      1. Download the subtitle WebVTT data, turn it into a string, use NSRegularExpression to parse the string and create an array of custom subtitle objects (each one containing a time range and the subtitle text for that time range)...
      2. Create an AVAsset from the remote URL of the video whose master playlist does NOT contain the WebVTT data, read from it using AVAssetReader...
      3. 300+ lines of reading and writing that won't even execute because...
      4. YOU CAN'T USE AVASSETREADER ON AN ASSET WITH A REMOTE URL!!!!!!!!!!
    2. At this point, I arrive at a similar (!!unverified!!) conclusion to bdanis in this post: I have to "download the m3u8 file locally, edit it by adding the vtt url to it, and [give] that local file url to the AVAsset".
    3. In comes AVAssetResourceLoaderDelegate. Using Apple's AVURLDelegateDemo sample code and another gist I found (see [1] at the bottom of the page) as a template, I:
      1. Create an AVURLAsset using a URL that is nearly identical to the reomte URL, but with a custom scheme (non http or https)...
      2. Create a class that conforms to AVAssetResourceLoaderDelegate and assign it to the asset's resourceLoader. Start loading the requested keys asyncronously...
      3. In "shouldWaitForLoadingOfResponse" (an AVAssetResourceLoaderDelegate method) I attempt to catch any resource request from the asset.
      4. In the same method, I lazily inititalize an NSURLConnection. This connection is initialzed with the ORIGINAL HTTP ADDRESS OF THE REMOTE VIDEO. The same object that conforms to AVAssetResourceLoaderDelegate also conforms to NSURLConnectionDataDelegate, so I initalize the NSURLConnection with that object in the "delegate" paramater as well...
      5. I hook into NSURLConnectionDelegate methods. In "connection:didReceiveData:" I write to a NSMutableData object that I have stored as a property. In "connectionDidFinishLoading:" I write the data to a path in the user's document directory...
      6. Using the file path from #3.5, I try to read from that path in my AVAssetReader from #1.3.
      7. None of the AVAssetResourceLoaderDelegate methods are being called, so I decided to write this post.


    • There is so little documentation regarding adding WebVTT to an m3u8 file client-side. No matter how much research I do regarding a certain strategy, it's dificult to know if it will work until I (1) get some AVFoundation-generated error, (2) google that error and (3) delete all of the test cases & code that I just wrote because, alas, you cannot "use AVMutableComposition for X" or "use AVAssetReader to read from Y".


    I can keep trying to refine my hybrid AVAssetReader/Writer + AVAssetResourceLoaderDeleagte apporach, but at this point I'd like to know if anyone - ANYONE - has tried something similar, has run into similar issues or has any sugestions. I haven't gotten the AVAssetResourceLoaderDelegate part (section C, #3) to work yet, and that is just so that I can have an AVURLAsset with a local URL so the other 300+ lines of the avsubtitleswriter-inspired reading & writing code (section C, #1) can run...god knows what runtime errors await me there.


    I'd greatly appreciate some feedback. Thanks for taking the time to read this!


    [1] https://gist.github.com/anonymous/83a93746d1ea52e9d23f

    Replies  13
    Boosts  0
    Views  11k
    Participants  8
    ishahak OP
    Nov ’16

    Hi @CuriosityStreamiOS, just wondering - did you ever get any progress with that?

    0
    JamesScott OP
    Nov ’16

    Hi @CuriosityStreamiOS Do you have any solution for this? I have same problem.

    0
    Media Engineer OP
    Apple
    Dec ’16

    I don't why you are insisting on doing it client side. Making a bunch of new master playlists isn't that much work. And the server they would sit on wouldn't get that much traffic from them. But since you've talked yourself into it...


    I would use AVAssetResourceLoaderDelegate. You only need to intercede in the master playlist requests. Your master playlist URLs would have a custom URL scheme. When you get one of these you do fetch the real master playlist, edit the result to include the WebVTT, and return that to the caller.


    Editing the master playlist shouldn't be that bad. You need to add EXT-X-MEDIA tags for each WebVTT and add SUBTITLE attributes to the EXT-X-STREAM-INF so the WebVTT gets picked up. The AVARLDelegateDemo sample code should help. Look in APLCustomAVARLDelegate.m

    0
    JamesScott OP
    Feb ’17

    Sometimes it is hard to convince the customer to change his playlist and he insists on solution on the client side. Thanks for answer, it helped me .

    0
    CuriosityStreamiOS OP
    Mar ’17

    @JamesScott apologies for the long delay in my response.


    You are absolutely spot on; our backend engineers insisted for months that we could do this client side. It took me several weeks to conclude that - in our case - doing this kind of client side playlist manipulation was too much. I took the opportunity to brush up on my PHP and dove into our backend to manipulate the playlists from there.


    Very glad you found a solution though!


    Also, @sw_mechanic: thanks for your reply!

    0
    jbweimar OP
    Jan ’19

    I wrote a small blog post with a link to a repository with working code for whoever is interested.


    https://jorisweimar.com/programming/supporting-external-webvtt-subtitles-in-avplayer/

    0
    sl56 OP
    May ’19

    Thanks for writting the blog, but the blog can not be opened. I was looking at your github as well, but with apple's sample m3u8, but it always throw AVFoundationErrorDomain Code=-11800. Any thought?

    0
    sl56 OP
    May ’19

    I used AVAssetResourceLoaderDelegate and only intercede in the master playlist requests with custom schema "fakehttp". Then I used urlsession to get th real master playlist , and then call


    loadingRequest.dataRequest?.respond(with: data)

    loadingRequest.finishLoading()


    however i am getting unknown error

    Do I need to change all fragment m3u8 to absolute path?

    0
    MXCat OP
    Jan ’21
    @jbweimar, your site is down 😢
    0
    kanderson@wellbeats.com OP
    Mar ’21
    i'm another poor soul going down this road, and i haven't yet gotten this to actually work, but i wanted to remedy the dead link above - jbweimar's site is down, but the code is here: https://github.com/jbweimar/external-webvtt-example

    another code sample is here: https://github.com/jaredsinclair/sodes-audio-example which has an accompanying blog post with helpful explanation (blog post is linked from the github post)

    this comment is also helpful: https://github.com/google/shaka-packager/issues/575#issuecomment-534471091
    0
    kanderson@wellbeats.com OP
    Mar ’21
    actually this is going very badly. avoid this if you can. i can't get it to work in new and exciting ways nobody else mentioned.
    0
    kanderson@wellbeats.com OP
    Apr ’21
    The github links I posted earlier turned out to not be as helpful as I thought. I'd like to remove those posts but I can't. All in all this was a horrible experience but I did succeed. My full explanation is too long to post here so I've stuck my long explanation and sample code on Github: https://github.com/kanderson-wellbeats/sideloadWebVttToAVPlayer . I dearly hope my link doesn't die. Apple doesn't want us posting long answers to questions but they're just fine forcing us to do weeks of ridiculous work for something that on another platform is an hour or less.
    0
    kanderson@wellbeats.com OP
    Apr ’21
    I've also posted my solution on StackOverflow: https://stackoverflow.com/a/67308098/15788389. Hopefully one of these links spans the ages.
    0
    Adding Subtitles From SEPARATE WebVTT File to AVAsset/PlayerItem
    First post date Last post date
     
     
    Q