0 Replies
      Latest reply on Apr 21, 2020 9:29 PM by oneStepAtATime
      oneStepAtATime Level 1 Level 1 (0 points)

        I am following the sample code at

        https://github.com/robovm/apple-ios-samples/tree/master/AVCustomEdit/AVCustomEdit

        The stitching of videos and transition works fine.

        I want to update the code to implement picture in picture video use case (maintaining use of APLCustomVideoCompositionInstruction design).

        Any guidance on this will be much appreciated.

         

        I have updated the class with 2 new functions: setPassThroughTrackID() and setTransitionWithSourceTrackIDs().

             - (void)setPassThroughTrackID:(CMPersistentTrackID)passthroughTrackID forTimeRange:(CMTimeRange)timeRange 
             { 
                  _passthroughTrackID = passthroughTrackID;
                  _requiredSourceTrackIDs = nil; 
                  _timeRange = timeRange; 
                  _containsTweening = FALSE; 
                  _enablePostProcessing = FALSE; 
                  return; 
             }  
        
             - (void)setTransitionWithSourceTrackIDs:(NSArray *)sourceTrackIDs forTimeRange:(CMTimeRange)timeRange
             { 
                  _requiredSourceTrackIDs = sourceTrackIDs; 
                  _passthroughTrackID = kCMPersistentTrackID_Invalid; 
                  _timeRange = timeRange; 
                  _containsTweening = TRUE; 
                  _enablePostProcessing = FALSE; 
                  return; 
             } 

        Here is a sample code for creating a composition instruction for normal stitching:

         

          sourceClips = [[NSMutableArray alloc] initWithCapacity:count];
          // Here creating a list of AVAsset using URL.
          [sourceClips addObject:[AVAsset assetWithURL:pUrlString1]];
          [sourceClips addObject:[AVAsset assetWithURL:pUrlString2]];
        
          AVMutableComposition *composition = [AVMutableComposition composition];
          NSMutableArray* compositionVideoTracks = [NSMutableArray array];
        
          CMTimeRange *passThroughTimeRanges = (CMTimeRange*)alloca(sizeof(CMTimeRange) * 2);
          CMTimeRange *srcClipTimeRanges = (CMTimeRange*)alloca(sizeof(CMTimeRange) * 2);
          CMTime *dstClipTimeStamp = (CMTime*)alloca(sizeof(CMTimeRange) * 2);
        
          int clipID = 0;
          passThroughTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          srcClipTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          dstClipTimeStamp[clipID] = CMTimeMake(0, 1);
        
          AVMutableCompositionTrack* track = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
          [compositionVideoTracks addObject:track];
        
          AVURLAsset *asset = [_clips objectAtIndex:clipID];
          AVAssetTrack *clipVideoTrack = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
        
          [compositionVideoTracks[clipID] insertTimeRange:srcClipTimeRanges[clipID] ofTrack:clipVideoTrack atTime:dstClipTimeStamp[clipID] error:&error];
        
          AVMutableCompositionTrack * track0 = compositionVideoTracks[clipID];
          APLCustomVideoCompositionInstruction *videoInstruction = [APLCustomVideoCompositionInstruction alloc];
          [videoInstruction setPassThroughTrackID:track0.trackID forTimeRange:passThroughTimeRanges[clipID]];

         

         

        For PIP I tried to give same timerange for 2 video tracks, but the output is not created.

        I am hard coding the time durations for 4s.

         

          sourceClips = [[NSMutableArray alloc] initWithCapacity:size];
          // Here I am adding AVAsset
          [sourceClips addObject:[AVAsset assetWithURL:pUrlString1]];
          [sourceClips addObject:[AVAsset assetWithURL:pUrlString2]];
        
          AVMutableComposition *composition = [AVMutableComposition composition];
          NSMutableArray* compositionVideoTracks = [NSMutableArray array];
        
          CMTimeRange *passThroughTimeRanges = (CMTimeRange*)alloca(sizeof(CMTimeRange) * 2);
          CMTimeRange *srcClipTimeRanges = (CMTimeRange*)alloca(sizeof(CMTimeRange) * 2);
          CMTime *dstClipTimeStamp = (CMTime*)alloca(sizeof(CMTimeRange) * 2);
        
          int clipID = 0;
          passThroughTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          srcClipTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          dstClipTimeStamp[clipID] = CMTimeMake(0, 1);
        
          AVMutableCompositionTrack* track = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
          [compositionVideoTracks addObject:track];
        
          AVURLAsset *asset = [sourceClips objectAtIndex:clipID];
          AVAssetTrack *clipVideoTrack1 = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
        
          [compositionVideoTracks[clipID] insertTimeRange:srcClipTimeRanges[clipID] ofTrack:clipVideoTrack1 atTime:dstClipTimeStamp[clipID] error:&error];
        
          AVMutableCompositionTrack * track0 = compositionVideoTracks[clipID];
          APLCustomVideoCompositionInstruction *videoInstruction = [APLCustomVideoCompositionInstruction alloc];
        
          // trying to add pip
          {
          clipID = 1;
          passThroughTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          srcClipTimeRanges[clipID] = CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1));
          dstClipTimeStamp[clipID] = CMTimeMake(0, 1);
        
          AVMutableCompositionTrack* trackPip = [composition addMutableTrackWithMediaType:AVMediaTypeVideo preferredTrackID:kCMPersistentTrackID_Invalid];
          [compositionVideoTracks addObject:trackPip];
          AVURLAsset *asset = [sourceClips objectAtIndex:clipID];
          AVAssetTrack *clipVideoTrack2 = [[asset tracksWithMediaType:AVMediaTypeVideo] objectAtIndex:0];
        
          [compositionVideoTracks[clipID] insertTimeRange:srcClipTimeRanges[clipID] ofTrack:clipVideoTrack2 atTime:dstClipTimeStamp[clipID] error:&error];
        
          AVMutableCompositionTrack * track1 = compositionVideoTracks[clipID];
          }
          videoInstruction.foregroundTrackID = track0.trackID;
          videoInstruction.backgroundTrackID = track1.trackID;
          [videoInstruction setTransitionWithSourceTrackIDs:@[[NSNumber numberWithInt:track0.trackID], [NSNumber numberWithInt:track1.trackID]] forTimeRange:CMTimeRangeMake(kCMTimeZero, CMTimeMake(4, 1))];