-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Message ingestion stops #15
Comments
We need to increase the performance of processing messages, as the current performance is only efficient for a small amount of messages flowing on the network. When the traffic becomes high (testing on testnet -> continuously running on mainnet), the process rate is lower than the receiving rate so the message queue grows indefinitely. We explore the options to create separate runtimes or threads to allow concurrent processes. Tokio Runtime
Threads
When to use which?
|
Agreed we need to ensure that performance isn't a bottleneck, but is there any reason to believe this is what is causing message ingestion to stop? |
If I run two listener radio in parallel (one subscribed to testnet and one to mainnet), ingestion doesn't stop for testnet where there's significantly less traffic than mainnet. This doesn't necessarily mean performance is the cause of this issue, more so it could lead to a situation where the operator becomes occupied with receiving new messages and has no availability to processing messages into the DB |
After periodic network reconnectio & added timestamps messages do not stop when running natively, but in Docker the messages still stop
The text was updated successfully, but these errors were encountered: