How to rotate ca cert with new root and --force

[nonessentialprototype]

I am on rke2 2.24.17, and I am trying to rotate the ca cert with a newly generated ca certs. We want to invalidate the old certs/kube config so we do NOT want it cross signed.

I've followed the step here modified for my needs and am running this on the first server
https://docs.rke2.io/security/certificates#rotating-custom-ca-certificates

mkdir -p /tmp/rke2/server/tls
cp  /var/lib/rancher/rke2/server/tls/service.key /tmp/rke2/server/tls

DATA_DIR=/tmp/rke2 PRODUCT=rke2 ./generate-custom-ca-certs.sh

rke2 certificate rotate-ca --path=/tmp/rke2/server --server=https://127.0.0.1:9345 --force

The rotate is successfully but after I run systemctl restart rke2-server.service the server never comes back up and and logs show

CA cert validation failed: Get "https://127.0.0.1:9345/cacerts" tls: failed to verify certificate x509 certificate signed by unknown authority.

This section also mentions

If you used the --force option or changed the root CA, ensure that any nodes that were joined with a secure token are reconfigured to use the new token value, prior to being restarted. The token may be stored in a .env file, systemd unit, or config.yaml, depending on how the node was configured during initial installation.

which seems to be a copy paste error cause the generate-custom-ca-certs does not output a new token.

I then tried these steps modifying the rotate script to not copy and use the old ones.
https://docs.rke2.io/security/certificates#rotating-self-signed-ca-certificates
with the same result, it rotates successfully but 509s on restart.

[brandond]

generate-custom-ca-certs does not output a new token.

yes, generate-custom-ca-certs was intended to be used before initial cluster startup. Using that that to switch from the default self-signed certs, to custom certs, is not something that we've tested. You'll need to calculate the hash yourself, and update the hash in the token files by hand.

[nonessentialprototype]

If that is the case the documentation here https://docs.rke2.io/security/certificates#using-the-example-script-1
, and the script should be updated as it even prints out

  echo "To update certificates on an existing cluster, you may now run:"
  echo "    k3s certificate rotate-ca --path=${DATA_DIR}/server"

[brandond]

Yes, it prints that because it is intended to also be used to update the custom certs when they need to be renewed. It is just use of that script to switch from default to custom that isn't a tested path.

[nonessentialprototype]

I have been able to rotate successfully once, subsequent attempts after all the nodes are up fail.

I ran across this comment k3s-io/k3s#8952 (comment)

Which got me thinking to modify the rotate-default-ca-certs.sh

Adding an If around ${TYPE}-ca.crt creation

if [[ $TYPE = "client" ]]; then
  cat ${TYPE}-ca.pem \
  ${TYPE}-root-ssigned.pem > ${TYPE}-ca.crt

else
    cat ${TYPE}-ca.pem \
      ${TYPE}-root-ssigned.pem \
      ${TYPE}-root-xsigned.pem \
      ${TYPE}-root-old.pem > ${TYPE}-ca.crt
fi

then running

PRODUCT=rke2 ./rotate-default-ca-certs.sh
rke2 certificate rotate-ca --path=/var/lib/rancher/rke2/server/rotate-ca --server=https://127.0.0.1:9345 --force
rke2 certificate rotate
systemctl restart rke2-server

once up, updating the config on each server/agent to the new token and then rebooting them.

Once they are all rebooted, the original kubeconfig has become invalidated.