I am writing a RTSP client with Flutter SDK that is also using the Apple VideoToolKit API for the HW decoder. I will refrain from posting code at the moment because I think it would just be easier to explain.
I'm using wireshark to inspect contents and it seems parsing is working correctly. My problem is I can't get the data in the right format for the decoder, hence getting OSStatus (Swift) error code -8969. If someone could please clarify the AVCC format along with the inner payload format expected by the decoder, that would be great.
My decoding is done for one frame at a time. So for each frame I create a new decoding session.
The SPS and PPS value is set to a static value on program startup, and then updated once the server begins sending RTP. I don't parse the sprop-parameter-sets value at the moment, I will add this at a later time.
The below buffer will be the resulting AVCC format for a FU-A RTP payload with 3 slices. Please let me know if there is anything I get wrong here. The 4 byte length is big endian representation. When I create the decoding session in Swift, I consider all of this to be 1 sample.
[4 byte length][FU identifier slice 1][FU header slice 1][NAL Unit payload slice 1][4 byte length][FU identifier slice 2][FU header slice 2][NAL Unit payload slice 2][4 byte length][FU identifier slice 3][FU header slice 3][NAL Unit payload slice 3]
The length is:
length = RTSP length field - RTP_HEADER_LEN
Where the RTP_HEADER_LEN is equal to 12 bytes.
Any guidance appreciated, thank you.
Update: I removed the FU header/FU identifier from the buffer, and am now getting 2 successful decodes out of hundreds of frames, but am still getting OSStatus codes: -666, -8969. and -12348. If anyone could explain any of these to me that would be helpful. Thanks.
