Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Subscription: payload size control for poll request & apply forced tablet for dataset topic & implement strictness timeout by multiple poll #13037

Merged
merged 80 commits into from
Aug 27, 2024

Conversation

VGalaxies
Copy link
Contributor

@VGalaxies VGalaxies commented Jul 26, 2024

Compatibility Issue Notice: If a client that subscribed prior to this PR uses the server after this PR, issues will arise when consuming.

This PR includes the following content:

  • payload size control for poll request
    • massive topics payload control on broker
    • support tablets streaming transmission
  • subscription-side memory control optimization
    • apply forced tablet for dataset topic to avoid failure of memory control
    • fix byte buffer size estimation
  • implement strictness timeout by multiple poll (default to one)
    • the throughput of consumption can be improved when the production speed is fast enough
  • some other refactoring
    • per batch per region for single prefetching queue
    • optimize prefetching queue resource release issue when closing
    • lazily remove committed events and recycle pollable events
    • support single consumer simultaneously pull multiple tsfiles

@SteveYurongSu SteveYurongSu self-assigned this Jul 26, 2024
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Jul 29, 2024
@VGalaxies VGalaxies marked this pull request as ready for review August 26, 2024 02:05
@VGalaxies VGalaxies changed the title Subscription: payload size control for poll request Subscription: payload size control for poll request & apply forced tablet for dataset topic & implement strictness timeout by multiple poll Aug 26, 2024
@SteveYurongSu SteveYurongSu merged commit 08f7cd8 into apache:master Aug 27, 2024
26 checks passed
@VGalaxies VGalaxies deleted the improve-massive-poll branch August 27, 2024 04:15
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Aug 27, 2024
…blet for dataset topic & implement strictness timeout by multiple poll (apache#13037)
VGalaxies added a commit to VGalaxies/iotdb that referenced this pull request Aug 27, 2024
…blet for dataset topic & implement strictness timeout by multiple poll (apache#13037)
SteveYurongSu pushed a commit that referenced this pull request Aug 27, 2024
…blet for dataset topic & implement strictness timeout by multiple poll (#13037) (#13309)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants