I encountered the same problem, and finnaly I found the reason behind the problem after varied tries, maybe useful for you. In my case, if the VideoToolbox encoder receives continuous two frames at least with the same pts, the encoder will decline output bitrate very quickly.. it's ok: pts: 0, 0.2, 0.4, 0.6, 0.8 .... it's bad (two frame with the same pts 0.4) pts: 0, 0.2, 0.4, 0.4, 0.6, 0.8 .... You may check pts of the frames passed to the encoder whether or not the same, if so, discard the later one or update the pts of the later.

Thanks for you reply. I found the solution yestoday, after giving duration value for every video frame's CMSampleTimingInfo struct, this problem disappeared. ///// Code with Problem  CMSampleTimingInfo timingInfo;  timingInfo.presentationTimeStamp = CMTimeMake(pts_ms, 1000);  timingInfo.decodeTimeStamp = CMTimeMake(dts_ms, 1000);  size_t sampleSize[1] = { frame_size };  CMSampleBufferRef sampleBuffer;  status = CMSampleBufferCreate(kCFAllocatorDefault,                 dataBlock,                 true, // dataReady                 NULL, // makeDataReadyCallback                 NULL, // makeDataReadyRefcon                 _videoFormatDescription,                 1, // numSamples                 1, // numSampleTimingEntries                 &timingInfo, // CMSampleTimingInfo *sampleTimingArray                 1, // numSampleSizeEntries                 sampleSize, // sampleSizeArray                 &sampleBuffer); //sampleBufferOut ///// Code without Problem  CMSampleTimingInfo timingInfo;  timingInfo.presentationTimeStamp = CMTimeMake(pts_ms, 1000);  timingInfo.decodeTimeStamp = CMTimeMake(dts_ms, 1000); // ⬇️⬇️⬇️ timingInfo.duration = CMTimeMake(duration, 1000)  size_t sampleSize[1] = { frame_size };  CMSampleBufferRef sampleBuffer;  status = CMSampleBufferCreate(kCFAllocatorDefault,                 dataBlock,                 true, // dataReady                 NULL, // makeDataReadyCallback                 NULL, // makeDataReadyRefcon                 _videoFormatDescription,                 1, // numSamples                 1, // numSampleTimingEntries                 &timingInfo, // CMSampleTimingInfo *sampleTimingArray                 1, // numSampleSizeEntries                 sampleSize, // sampleSizeArray                 &sampleBuffer); //sampleBufferOut

I have tried: 1.both the pull style and push style API, neither works well. I use serial NSOperationQueue to make sure all writer operation be serial in push sytle, doesn't help too 2._writer.shouldOptimizeForNetworkUse set to be YES or NO, neither helps. When YES is set, there's no file after finishing writing, when NO is set, there's a mp4 file without moov atom, only 'ftyp', 'wide' and a broken 'mdat' atom exist