-
Notifications
You must be signed in to change notification settings - Fork 65
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow modification of configs if the first pod of rack fails #583
Comments
Why would we need the rest of the pods to be up? |
We need to be able to detect what a failed state would be for a pod in the statefulset (it could be pending scheduling, or having wrong coordinates for the image, etc...). If the change was applied to a single pod (the revision of the other pods is still the previous one) and the last pod isn't in "Running" state (it's the first updated one), then we allow a new update to the statefulset in order to fix what was preventing the pod from starting. |
What is missing?
Currently, if user deploys something that's broken and causes the first pod of the rack to not start, we require the user to manually set "forceUpgradeRacks" property which will then override the current configuration despite the datacenter not being ready from previous change.
This process feels like something we could automate. If we're still processing the first pod (but rest are up), we should probably allow user to fix the configuration and then we would apply it to make the cluster healthy again. In other words, try to detect when "forceUpgradeRacks" is going to be required and just simply apply it in our internal process.
Even better would be to rollback to previous setting, but we don't want to downgrade the user set CRDs by ourselves as that could mess up with tools like Argo.
Why is this needed?
Automatic recovery is not easy if it requires modifying the CRD and then cass-operator will modify the CRD to ensure next time it's not applicable. We want to simplify the user experience by automatically detecting this issue as these incorrect settings do appear every now and then (user mistakes happen).
┆Issue is synchronized with this Jira Story by Unito
┆Fix Versions: 2024-10
┆Issue Number: CASS-17
The text was updated successfully, but these errors were encountered: