Adding audio track to PeerConnection

[yrucrem]

Hello!

I'm trying to send audio data over a peer connection and I'm not sure that my setup for that is correct. Both sides of the connection use libdatachannel for WebRTC. Establishing the peer connection works and I can send data over the datachannel, but when I add an audio track and then try to send data via Track::send() I just get warnings on the server side:

Number of RTP packets received with an unknown packet type: 61 (over 1 seconds)

When initializing the peer connection I made sure to set forceMediaTransport to true and I added an onTrack callback:

	peer_connection->onTrack([this](std::shared_ptr<rtc::Track> track) {
		printf("Track added\n");
		// ...
	});

The code to add the track looks like this:

	rtc::Description::Audio audio("audio", rtc::Description::Direction::SendOnly);

	audio.addSSRC(next_ssrc++, "audio-send");

	std::shared_ptr<rtc::Track> track = peer_connection->addTrack(audio);

where next_ssrc is a running counter. The created track is put into a map and when audio data is available I create a std::byte array and send it with

	track->send(data, data_len);

I noticed that I do not see the "Track added" message on the remote peer connection, so I'm not sure if the remote peer is even aware that the track was added. And when I start sending data to the track, I get the messages about unknown packet types on the other end.

Am I missing a step in setting up the connection?

Any hints and tipps on how to establish an audio connection between two libdatachannel peer connections would be greatly appreciated!

Thanks in advance and best regards,
Patrick

[paullouisageneau]

There are a few things missing:

• You need to declare the format of your audio data with an RTP payload type in the description. For instance, if it is Opus-encoded:

audio.addOpusCodec(96);

• If your audio stream is not already packetized as RTP, you must configure the track to packetize your audio stream. For instance, for Opus:

auto rtpConfig = make_shared<RtpPacketizationConfig>(ssrc, "audio-send", 96);
auto packetizer = make_shared<OpusRtpPacketizer>(rtpConfig);
track->setMediaHandler(packetizer);

• After each send, you must increase rtpConfig->timestamp by the audio duration in samples. You should send audio in chunks matching the audio frame duration (20ms by default), so for Opus at 48000Hz in would be an increase of 960. Alternatively you can also use RtpPacketizationConfig::secondsToTimestamp() helper function. (I'm planning to introduce a sendFrame() method that would do that for you provided you send the expected duration.)

[yrucrem]

Hello again!

As you suggested I checked again, if user code was blocking the callback and you were right, the callback is called, but on a windows build the console output was suppressed, so I just did not see the messages and I could fix that now.

So I am a step further and get the remote track that was added on the server, thank you again for all your input!

Unfortunatelly I ran into a similar problem, when I try to add a track on both sides of the peer connection (one to send audio data from the server to the client and the other one for the reverse direction): In this case I do not get any calls to the onTrack() callbacks on both sides.

The process is as follows:

1. Server creates peer connection.
2. Server adds audio track (ssrc=2, cname="audio-stream")
3. Server creates datachannels.
4. Server calls setLocalDescription()
5. Server sends created offer to client via signaling connection.
6. Client creates peer connection.
7. Client adds audio track (ssrc=3, cname="audio-stream")
8. Client sets offer as remote description.
9. Client sets generated answer as local description.
10. Client sends created answer to server.
11. Server sets answer as remote description.

The disableAutoNeogotiation and forceMediaTransport settings are true on both sides.

This is the generated offer:

v=0
o=rtc 2262753625 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio 0
a=group:LS audio
a=msid-semantic:WMS *
a=setup:actpass
a=ice-ufrag:4HUi
a=ice-pwd:****
a=ice-options:ice2,trickle
a=fingerprint:sha-256 ****
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=mid:audio
a=sendonly
a=ssrc:2 cname:audio-stream
a=rtcp-mux
a=rtpmap:96 opus/48000/2
a=fmtp:96 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:0
a=sendrecv
a=sctp-port:5000
a=max-message-size:262144

and this the generated answer:

v=0
o=rtc 1109582792 0 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE audio 0
a=group:LS audio
a=msid-semantic:WMS *
a=setup:active
a=ice-ufrag:K7zj
a=ice-pwd:***
a=ice-options:ice2,trickle
a=fingerprint:sha-256 ***
m=audio 9 UDP/TLS/RTP/SAVPF 96
c=IN IP4 0.0.0.0
a=mid:audio
a=sendonly
a=ssrc:3 cname:audio-stream
a=rtcp-mux
a=rtpmap:96 opus/48000/2
a=fmtp:96 minptime=10;maxaveragebitrate=96000;stereo=1;sprop-stereo=1;useinbandfec=1
m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=mid:0
a=sendrecv
a=sctp-port:5000
a=max-message-size:262144

I would expect the answer to contain both audio-tracks not just the one created on the client. When only the server adds an audtio-track, I get the log message about the received audio-track directly after setting the offer as remote description.

I suspect that I am doing something I am not supposed to or have the order of the steps mixed up. That's why I added my steps in such detail (please excuse the long post, I did not want to miss any important information).

Do you maybe see any problems with the taken steps and the created session descriptions?

Thanks again for your help and any further assistance, it is greatly appreciated!

Best regards,
Patrick

[paullouisageneau]

You can't add a track in an answer by specification, only an offer can contain new tracks. You would need a renegotiation to add a track from the other side.

However, you don't need two tracks in this case, simply create one track with sendrecv direction. The client will need to add the SSRC it is going to send in onTrack():

peer_connection->onTrack([](std::shared_ptr<rtc::Track> track) {
    auto description = track->description();
    // [You can check here what direction, payload type, and format the other side offers]
    description.addSSRC(ssrc, "audio-send");
    track->setDescription(description);
});

In the general case, a single offer is sufficient provided no track is added after connection, as you can also offer to receive only.

[yrucrem]

Ah thank you very much for the clarification, I was suspecting that somehting was not quite right in my usage of the library.

Is there anything special I need to do on the tracks media handlers when I want to send data in both directions? You made me aware, that I need to add the Opus RTP packetizer via setMediaHandler(), Is this also necessary when receiving the track in the onTrack() callback (with a new rtpConfig containing the other SSRC)?

In the media-receiver example I see that the added media handler is a RtcpReceivingSession and I am not sure how to set both handlers.

If I am not mistaken there is no example where a media track is used both for sending and receiving, could you maybe provide a simple example? Or if I just overlooked it point me to an example that covers this topic?

Thanks for your continued support, it really helps a lot to understand this topic better.

Best regards,
Patrick

[paullouisageneau]

Is there anything special I need to do on the tracks media handlers when I want to send data in both directions? You made me aware, that I need to add the Opus RTP packetizer via setMediaHandler(), Is this also necessary when receiving the track in the onTrack() callback (with a new rtpConfig containing the other SSRC)?

Yes indeed, if you send the same format in both directions, you need to set up the track the same way in onTrack().

In the media-receiver example I see that the added media handler is a RtcpReceivingSession and I am not sure how to set both handlers.

If I am not mistaken there is no example where a media track is used both for sending and receiving, could you maybe provide a simple example? Or if I just overlooked it point me to an example that covers this topic?

There is no example for bidirectional tracks, but the way you set multiple handlers is the same as for unidirectional tracks, only you need to do it both on addTrack() and onTrack(). Handlers may be chained together to perform the processing you want. For instance in your case, you should do something like:

  auto rtpConfig = make_shared<RtpPacketizationConfig>(ssrc, "audio-send", 96);
  auto handler = std::make_shared<rtc::OpusRtpPacketizer>(rtpConfig); // packetize outgoing
  handler->addToChain(std::make_shared<rtc::OpusRtpDepacketizer>()); // depacketize incoming
  track->setMediaHandler(handler);

It will work like this but to make it complete you should also send RTCP reports:

  auto rtpConfig = make_shared<RtpPacketizationConfig>(ssrc, "audio-send", 96);
  auto handler = std::make_shared<rtc::OpusRtpPacketizer>(rtpConfig); // packetize outgoing
  handler->addToChain(std::make_shared<rtc::OpusRtpDepacketizer>()); // depacketize incoming
  handler->addToChain(std::make_shared<rtc::RtcpReceivingSession>()); // to send RTCP receiver reports
  handler->addToChain(std::make_shared<rtc::RtcpSrReporter>()); // to send RTCP sender reports
  track->setMediaHandler(handler);

To receive, you should use the onFrame() callback which gives you metadata like the timestamp contrary to onMessage():

  track->onFrame([](const rtc::binary data, rtc::FrameInfo info) {
	// handle the data
  });

And thank you so much for the ko-fi!

[yrucrem]

Thank you very much, that helped a lot to clear things up! I will try that as soon as I can and give an update if I got it to work.

And thank you so much for the ko-fi!

You're perfectly welcome, it feels like the least I could do. :-) I appreciate your work on the library and your swift support very much!

[yrucrem]

Small update: I did my first tests, and the audio data is correctly transferred between client and server. Thanks again for your support, it helped a lot for understanding libdatachannel und WebRTC intricacies!

[paullouisageneau]

Great, you are welcome!