Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP fix: add delay to prevent starting parallel activations #924

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

Alex-Izquierdo
Copy link
Collaborator

@Alex-Izquierdo Alex-Izquierdo commented Jun 3, 2024

When we have multiple workers (the usual scenario) and we create or enable a batch of activations, the activations are scheduled and processed in parallel causing sometimes a race condition. The race condition has two potential effects:

  • If all the nodes are idle, the dispatcher can schedule the activations in the same node, reaching the "max_activations_limit" and postponing some of them. This loop can happen multiple times depending on the amount of activations and workers.
  • We create more activations than the allowed by the max_limit_activations.

As per internal discussions, add a random delay when scheduling start operation to break the potential parallelism.

@Alex-Izquierdo Alex-Izquierdo changed the title fix: add delay to prevent starting parallel activations WIP fix: add delay to prevent starting parallel activations Jun 3, 2024
Signed-off-by: Alex <aizquier@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant