Message ingestion stops

[hopeyen]

After periodic network reconnectio & added timestamps messages do not stop when running natively, but in Docker the messages still stop

[hopeyen]

We need to increase the performance of processing messages, as the current performance is only efficient for a small amount of messages flowing on the network. When the traffic becomes high (testing on testnet -> continuously running on mainnet), the process rate is lower than the receiving rate so the message queue grows indefinitely.

We explore the options to create separate runtimes or threads to allow concurrent processes.

Tokio Runtime

• Fast and scalable network applications.
• Event-driven, non-blocking I/O model using the async/await syntax - instead of threads being blocked waiting for I/O operations, the event loop can handle other tasks.
• Tasks spawn asynchronously are more lightweight than threads. Multiple tasks can run on a single OS thread.
• Asynchronous integrated I/O operations including TCP/UDP networking, filesystem operations, and timers.

Threads

• General-purpose concurrent and parallel programming, especially for CPU-bound tasks.
• More traditional method of achieving concurrency and parallelism on the OS-level.
• Effective for CPU-bound tasks utilizing multiple cores; less efficient than asynchronous tasks for I/O-bound operations due to context switching.
• Each thread has its own stack which can add up when spawning many threads, leading to high memory usage.

When to use which?

• I/O-bound tasks such as a web server or a database, Runtime might be more efficient because of its non-blocking I/O model.
• CPU-bound tasks for performing computations that are CPU-bound using multiple cores.

[chriswessels]

Agreed we need to ensure that performance isn't a bottleneck, but is there any reason to believe this is what is causing message ingestion to stop?

[hopeyen]

but is there any reason to believe this is what is causing message ingestion to stop?

If I run two listener radio in parallel (one subscribed to testnet and one to mainnet), ingestion doesn't stop for testnet where there's significantly less traffic than mainnet. This doesn't necessarily mean performance is the cause of this issue, more so it could lead to a situation where the operator becomes occupied with receiving new messages and has no availability to processing messages into the DB

[hopeyen]

Can confirm that listener radio message ingestion is now continuous after

• making a separate runtime for processing messages: a429546
• periodically prune database: 0ce8b5b
• deployed and monitored for 1 day on launchpad