Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Rework decompression algorithm to operate over 4-byte chunks #45

Draft
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

NickCondron
Copy link
Contributor

I tried implementing the chunk decompression approach (used in the original library) in Rust. This is very WIP, but I was curious what you thought about this idea. It's my first time writing Rust with this much unsafe, so I'm open to ideas to improve how it's organized.

I used the benchmark code in the my other PR. The performance results are mixed but promising. Maybe my code could be further optimized. Maybe it's worth testing other data sets.

dbtext/wikipedia/decompress
                        time:   [1.3307 ms 1.3312 ms 1.3319 ms]
                        thrpt:  [1.1472 GiB/s 1.1478 GiB/s 1.1482 GiB/s]
                 change:
                        time:   [+11.683% +11.794% +11.883%] (p = 0.00 < 0.05)
                        thrpt:  [-10.621% -10.550% -10.461%]
                        Performance has regressed.
Found 12 outliers among 100 measurements (12.00%)
  1 (1.00%) low severe
  1 (1.00%) low mild
  4 (4.00%) high mild
  6 (6.00%) high severe

dbtext/l_comment/decompress
                        time:   [606.27 ┬╡s 606.73 ┬╡s 607.24 ┬╡s]
                        thrpt:  [1.5616 GiB/s 1.5629 GiB/s 1.5641 GiB/s]
                 change:
                        time:   [-12.526% -12.458% -12.392%] (p = 0.00 < 0.05)
                        thrpt:  [+14.145% +14.230% +14.320%]
                        Performance has improved.
Found 15 outliers among 100 measurements (15.00%)
  4 (4.00%) high mild
  11 (11.00%) high severe

dbtext/urls/decompress  time:   [1.7961 ms 1.7986 ms 1.8027 ms]
                        thrpt:  [1.4759 GiB/s 1.4792 GiB/s 1.4812 GiB/s]
                 change:
                        time:   [-8.4988% -8.3049% -8.0170%] (p = 0.00 < 0.05)
                        thrpt:  [+8.7158% +9.0571% +9.2882%]
                        Performance has improved.
Found 13 outliers among 100 measurements (13.00%)
  1 (1.00%) low severe
  5 (5.00%) high mild
  7 (7.00%) high severe

@a10y
Copy link
Contributor

a10y commented Oct 11, 2024

Hey! Thanks for opening, this is definitely something we want to add. Apologies for being slow here, will find some time in next few days to get you some feedback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants