Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nsqd: switch to Klaus Post's flate compression library #1488

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

philpearl
Copy link

This is a dropin replacement for the standard library flate compression.

In addition to this I've also

  • cleaned up nsqd benchmarks and got them running (but only with -benchtime 1000x)
  • added a benchmark for compression

To really get the benefits of improving compression for go-nsq users we also need the same change in go-nsq. I've got a draft pull request here: nsqio/go-nsq#363

Similarly snappy performance needs nsqio/go-nsq#362

philpearl and others added 6 commits April 29, 2024 17:17
- Close sockets to ensure timely shutdown. This is the major improvement. Without this benchmarks would regularly time out
- Switch to a logger with no output to make the benchmark output readable
- Fix one unix socket benchmark to use unix socket opening function
- Fixed BenchmarkGUID to be a valid test - and added a similar TestGUID

With these changes I've also found I need to restrict b.N or NSQD takes too long to exit when N gets large. I've seen this in tests that publish only and do not receive messages. A large number of messages then need to be persisted and this takes tens of seconds.

Once these changes are made `go test -bench . -run ^$ -benchtime 1000x` completes cleanly for me.
This is a dropin replacement for the standard library flate compression.
@mreiferson mreiferson changed the title nsqd: switch to Klaus Post's flate compression library. nsqd: switch to Klaus Post's flate compression library May 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants