Values expiring with permanent=true

[ghenry]

On 2.4.0 should this be happening?

I'm getting all my own puts back off the DHT too, which I skip as I'm checking what put them there.

Thanks.

[ghenry]

This also results and in a big rush inbound and one of the opendht threads going 100% CPU.

[aberaud]

This looks like a catastrophic bug

• What's the lifetime of the nodes performing the put?
• How long do you wait before seeing that?

Thanks

[ghenry]

This looks like a catastrophic bug

• What's the lifetime of the nodes performing the put?

Ah, I'm stopping and starting this development node. What triggers it? A mode going offline?

• How long do you wait before seeing that?

Hmmm, I'll add a timer or something when I first see an expired value.

[ghenry]

So, my sentrypeer node has been up since 16:10 today and you can see that the event timestamps of bad_actors all created after that, so they shouldn't have expired:

You can prove that, because in this screenshot I don't process them as they have the same node_id (uuid) as the currently running node:

In this console screenshot you can see I'm saving them all permanently (https://github.com/SentryPeer/SentryPeer/blob/main/src/peer_to_peer_dht.c#L345):

Then they all flood in again:

When you see the flood of expires and flood of values for the bad_actors key, that thread CPU sits at 100%. Right now it's doing this. I could take a video if it helps :-)

Thanks.

[ghenry]

This is a vanillia dhtnode from master branch today, on the same box bootstrapped to bootstrap.sentrypeer.org up since 16:04 showing the same set of values coming in again via l bad_actors. Note the timestamps too:

and then I see them expire again on my sentrypeer node, then they go round and round :-)

[ghenry]

I've stopped all my nodes now and restarted, but things are probably living on other DHT nodes that I'm not running as per 4222 is open....let's see.

[aberaud]

Thanks. How many values are on this key approximately?

[aberaud]

Ah, I'm stopping and starting this development node. What triggers it? A mode going offline?

For the "permanent put" feature to work, the node would need to stay online for the duration of the value lifetime

[ghenry]

Ah, I'm stopping and starting this development node. What triggers it? A mode going offline?

For the "permanent put" feature to work, the node would need to stay online for the duration of the value lifetime

Yep, going by my screenshots the same node was.

[ghenry]

Thanks. How many values are on this key approximately?

No idea. Can I see that on dhtnode if I do a get?

[ghenry]

Just done that and pasted into a text file. 210 values at the moment.

[ghenry]

If there is connectivity issues too, I guess this can happen? Who is expiring these values? The server I bootstrapped to or other nodes? Just trying to understand. This node is running on broadband:

[aberaud]

I updated dhtcnode so it now does something similar to the C++ node.

Still investigating this issue. Some standalone code to reproduce the problem would be useful.

[aberaud]

I made a few tests on my side and had no issue with get, put and listen on this key using the C bindings and dhtnode.

[ghenry]

Have there been changes since I reported this then?

[aberaud]

Could you please try with 2.4.1 ?

I could reproduce some issues when the value count was reaching the value limit per key (for a given node). At least some of these issues should now be solved. The limit has also been raised from 1024 to 64k.

[ghenry]

Will do. I'll update Alpine and Homebrew too.

[aberaud]

A middle-click-paste introduced a typo in the 2.4.1 release commit -_-
Made a 2.4.2

[ghenry]

Is 2.4.2 coming out today? Just done - Homebrew/homebrew-core#99672

[ghenry]

Alpine submitted too.

[aberaud]

The tag is there, just the GitHub release is not documented yet

[ghenry]

Still seeing this, but my bootstrap node is on 2.4.0 via Homebrew. Test node gets restart often and is on 2.4.2. Need to get this 2.4.3 out so I can test on that.

Value callback expired: {"app_name":"sentrypeer","app_version":"1.4.1","event_timestamp":"2022-04-23 01:06:19.505170365","event_uuid":"f93e6871-2dd7-45fc-b1ac-526a575898ac","created_by_node_id":"04ec5965-4e4b-415a-b464-ecf1cc1fa90d","collected_method":"responsive","transport_type":"UDP","source_ip":"118.123.237.29","destination_ip":"xxx","called_number":"","sip_method":"","sip_user_agent":"","sip_message":""}

[ghenry]

Hi @aberaud

Any thoughts on this still? I'd really like to get this expired thing sorted and a bandwidth limiter in place for UDP traffic in the lib.

They flood in from peers as per https://twitter.com/ghenry/status/1534525326155554817 but

Thanks,
Gavin.

[aberaud]

The problem is that every sentrypeer dht node puts every value on the same key (bad_actors). The DHT Kademlia design distributes the load over the different keys. In case of significant load on a single key, best-effort applies and there is no guarantee to obtain all the values, instead, the load will spread on adjacent nodes as nodes taking too much traffic will stop responding (for BitTorrent, this means a popular torrent won't flood all the same nodes, and peers later exchange peer lists directly with PEX).

The best way to handle this might be to put values on the hash of the host to verify, and perform a get when needed to check this host. This would ensure the load of sentrypeer would better spread on the distributed network and greatly improve the reliability of both sentrypeer and the public opendht network, at the cost of added latency when verifying a new host.