Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Related to and discussed in #3004
This change renames
ZSTD_d_stableOutBuffer
toZSTD_d_outBufferMode
and adds a 3rd setting:The change is ABI-compatible.
Temporary source compatibility could be done by re-adding the original
to zstd.h until the parameter is no longer considered experimental.
--- What this does ---
ZSTD_bufmode_expose
gives a user direct read access into the LZ-window instead of flushing it with a memcpy.Instead of fiddling with an external buffer like the usual code:
we can now do this, completely avoiding an extra copy:
This is very nice for a streaming scenario where the consumer does not care about the exact size of the decompressed data and where the buffer is located in memory, as described in the ryg blog.
I guess avoiding the extra copy should also cause less cache thrashing but i didn't try to measure that.
Things left to discuss:
ZSTD_bufmode_expose
is currently only implemented for decompression.Since compression is quite resource-hungry this would not benefit from an exposed window. But it'd be nice to have the same API available for compression as well.
What about renaming
ZSTD_c_stable{In|Out}Buffer
toZSTD_c_outBufferMode
for consistency? (Didn't touch that so far because i wanted to avoid cluttering up the PR more that it is already.)Anything else left to do?
EDIT: Except for the C90 compat. Clang and vs19 are happy so i didn't see this. Oh well.
EDIT2: Fixed derp in the external buffer example block above, that was supposed to fwrite using pos, not size. In the exposed buffer block below, pos is always == 0. I considered setting output pos=size to ease a transition to this type of buffer handling but it caused some annoying side effects in the code that checks/handles pos so i didn't go there (mainly the pos<=size invariant, didn't want to touch that)
(cc @terrelln)