The WebSocket support via TLS cannot get past the handshake

[mpietras]

I followed the example and created a WebSocketServer like so:

	rtc::InitLogger(rtc::LogLevel::Info);

	// configure a websocket handler
	rtc::WebSocketServer::Configuration c;
	c.port = 8443;
	c.enableTls = true;
	c.certificatePemFile = "host.crt";
	c.keyPemFile = "host.key";
	auto webSocketServer = std::make_shared<rtc::WebSocketServer>(c);

	webSocketServer->onClient([](std::shared_ptr<rtc::WebSocket> client) {
		std::cout << "WebSocketServer: Client connection received" << std::endl;
		
		client->onOpen([wclient = std::weak_ptr(client)]() {
			std::cout << "WebSocketServer: Client connection open" << std::endl;
			if (auto client = wclient.lock())
				if (auto path = client->path()) {
					std::cout << "WebSocketServer: Requested path is " << *path << std::endl;
				}
			});

		client->onClosed([]() {
			std::cout << "WebSocketServer: Client connection closed" << std::endl;
		});

		client->onMessage([wclient = std::weak_ptr(client)](rtc::message_variant message) {
			if (auto client = wclient.lock()) {
				client->send(std::move(message));
			}
		});
	});

The library was compiled with OpenSSL on Ubuntu24.04. I verified that the host.crt and host.key files load correctly. Upon running, the output looks like:

INFO  [81853] [rtc::impl::WebSocketServer::runLoop@72] Starting WebSocketServer
WebSocketServer: Client connection received
WebSocketServer: Client connection closed
ERROR [81844] [rtc::impl::TlsTransport::doRecv@804] TLS handshake failed

I debugged into the library and it seems like it's coming out of Connecting state in tlstransport.cpp with a 2 which is SSL_ERROR_WANT_READ.

			if (state() == State::Connecting) {
				// Continue the handshake
				int ret, err;
				{
					std::lock_guard lock(mSslMutex);
					ret = SSL_do_handshake(mSsl);
					err = SSL_get_error(mSsl, ret);
					flushOutput();
				}

				if (openssl::check_error(err, "Handshake failed")) {
					PLOG_INFO << "TLS handshake finished";
					changeState(State::Connected);
					postHandshake();
				}
			}

I see that openssl::check_error ignores the error (which is correct since it just means there's more to do before handshake has finished) but then then since (state() != State::Connected) it drops around in the loop to the top where mIncomingQueue.running() is false then it ends with SSL_shutdown(mSsl);. So, I'm unsure if I'm utilizing the API incorrectly or if it's mishandling the transitory state during the handshake.

Any thoughts/direction is appreciated, thanks!

[paullouisageneau]

You must store client somewhere if you want to serve it, otherwise there are no more shared_ptr at the end of the callback, so the object is destroyed, the connection closed, and that's why the TLS handshake then aborts.

[mpietras]

What an oversight on my part - thank you for quickly spotting my simple issue and ending my way-too-deep search for the problem!