Sending AVPacket from ffmpeg library

[vulkan101]

I have cross posted this on S/O so apologies if you are reading it twice, I will try to keep them both up to date.

I'm encoding a video frame with the ffmpeg libraries, generating an AVPacket with compressed data.

Thanks to some recent advice on S/O, I am trying to send that frame over a network using libdatachannel, specifically by adapting the streaming example.

You will be able to tell I am quite new to streaming...

I am seeing problems inside h264rtppacketizer.cpp (part of the library, not the example) which are almost certainly to do with how I'm providing the sample data.
I don't think that this is anything to do with libdatachannel specifically, it will be an issue with what I'm sending..

I have naively copied the contents of an AVPacket->data pointer into the sample vector in the example:

sample.resize(pkt->size);
memcpy(sample.data(), pkt->data, pkt->size * sizeof(std::byte));    

but the packetizer is falling over when trying to get length values out of that data.
Specifically, in the following code, the first iteration gets a length of 1, but the second, looking up index 5, gives 1119887324. This is way too big for my data, which is only 3526 bytes (the whole frame is a single colour so likely to be small once encoded):

while (index < message->size()) {
assert(index + 4 < message->size());
auto lengthPtr = (uint32_t *)(message->data() + index);
uint32_t length = ntohl(*lengthPtr);
auto naluStartIndex = index + 4;
auto naluEndIndex = naluStartIndex + length;
assert(naluEndIndex <= message->size());	
		
auto begin = message->begin() + naluStartIndex;
auto end = message->begin() + naluEndIndex;
nalus->push_back(std::make_shared<NalUnit>(begin, end));
index = naluEndIndex;
}

Here is a dump of

uint32_t length = ntohl(*lengthPtr);

for the first few elements of the message (*lengthPtr in parentheses):

[2022-03-29 15:12:01.182] [info] index 0: 1  (16777216)
[2022-03-29 15:12:01.183] [info] index 1: 359  (1728118784)
[2022-03-29 15:12:01.184] [info] index 2: 91970  (1114046720)
[2022-03-29 15:12:01.186] [info] index 3: 23544512  (3225577217)
[2022-03-29 15:12:01.186] [info] index 4: 1732427807  (532693607)
[2022-03-29 15:12:01.187] [info] index 5: 1119887324  (3693068354)
[2022-03-29 15:12:01.188] [info] index 6: 3223313413  (98312128)
[2022-03-29 15:12:01.188] [info] index 7: 534512896  (384031)
[2022-03-29 15:12:01.188] [info] index 8: 3691315291  (1526728156)
[2022-03-29 15:12:01.189] [info] index 9: 83909537  (2707095557)
[2022-03-29 15:12:01.189] [info] index 10: 6004992  (10574592)
[2022-03-29 15:12:01.190] [info] index 11: 1537277952  (41307)
[2022-03-29 15:12:01.190] [info] index 12: 2701131779  (50331809)
[2022-03-29 15:12:01.192] [info] index 13: 768  (196608)

• I am fairly sure I am just missing something basic. E.g. am I supposed to do something with the AVPacket side_data, does AVPacket have or miss some header info?

• If I just fwrite the pkt->data for a single frame to disk, I can read the codec information with ffprobe:

Input #0, h264, from 'encodedOut.h264':
Duration: N/A, bitrate: N/A
Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1280x720, 30 tbr, 1200k tbn

• whereas the same for the example input files (again a single frame) gives the following:

[h264 @ 000001c88d1135c0] Format h264 detected only with low score of 1, misdetection possible!
[h264 @ 000001c88f337400] missing picture in access unit with size 85306
[extract_extradata @ 000001c88d11ee40] No start code is found.
sample-0.h264: Invalid data found when processing input

I would really appreciate any help this community can offer!

[paullouisageneau]

The streamer example input file use 32-bit length as NAL unit separator. Therefore, the H264RtpPacketizer instance is created with H264RtpPacketizer::Separator::Length.

If I'm not mistaken the ffmpeg output will have 4-byte start sequences as NAL unit prefix instead (which is actually more common), so if you change the packetizer setting to H264RtpPacketizer::Separator::LongStartSequence it should accept your sample.

[vulkan101]

@paullouisageneau the S/O question I posted is here, if you want to add the above as an answer, fire away!

[paullouisageneau]

Sure!

[vulkan101]

Thanks! that definitely seems to help - it isn't falling over any more anyway. The video isn't showing in the browser but it's progress!
I'll keep this discussion open if that's ok, in case I need to ask about the video display

[vulkan101]

So as expected, I could do with another pointer or two :-)
Symptom: no video displayed.
I am looking at the chrome diagnostics on chrome://webrtc-internals/, and in the example as provided, there is codec information for the RTCInboundRTPVideoStream_1 as below:

ssrc	1
kind	video
trackId	RTCMediaStreamTrack_receiver_39
transportId	RTCTransport_video-stream_1
codecId	RTCCodec_video-stream_Inbound_102
[codec]	H264 (102, level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f)
mediaType	video
jitter	0.004

and unsurprisingly there is a non-zero count for framesDecoded in the same stats.

When using the AVPacket based stream, the top of the same stats looks like this:

timestamp	30/03/2022, 17:52:51
ssrc	1
kind	video
trackId	RTCMediaStreamTrack_receiver_41
transportId	RTCTransport_video-stream_1
mediaType	video
jitter	0.004
packetsLost	0
packetsReceived	24510

framesDecoded is zero, with plenty of header bytes read.

This seems like Chrome doesn't recognise the codec and can't decode the frames.

I am encoding using ffmpeg codec name libx264 which translates to the enum AV_CODEC_ID_H264, and I set the option profile=baseline. If I write the frames to disk they can be read as a .H264 file by VLC.
Any clues regarding setting up the codec? Do I need to write some additional header info possibly?

to update, if I encode using h264_mf, I can display video, so I think this must be a setting on the libx264 encoder that is causing the problem.

[vulkan101]

Thanks! Will update here when I have news :-D

[swilcox3]

@vulkan101 Did you ever figure this out? I have a potentially similar issue, although I'm not sure if it's related to the codec or not: #699

[vulkan101]

@swilcox3 yeah I did, not near my machine right now but will try to update in the morning. In the meantime the discord channel is worth a look (most of my issues got worked out there tbh)

[vulkan101]

@swilcox3 I think actually I didn’t hit discord til a bit later. This is the S/O question I cross posted, with an answer: https://stackoverflow.com/questions/71660975/how-to-send-ffmpeg-avpacket-through-webrtc-using-libdatachannel

[swilcox3]

I actually saw that S/O question first, and I'm already using the LongStartSequence on the packetizer. I'll try Discord.