Need help when re-add a master node

[rogersollberger]

Hi,

I installed a cluster with 3 master nodes and some worker nodes (with cilium). Now one master node was damaged (vm).
Now when i try todo following steps, im not able to get master added again to the cluster.

kubectl drain --ignore-daemonsets --delete-emptydir-data
kubectl delete node

recreate the vm

the other two master i modified the config.yaml in a way that master01 is now the leading master and master02 points to master01
restarted them both.

install rke2 again on master00 but this time with config.yaml that he will join master01

but when i start the rke2-server it never joins to the rest.
Is there something wrong in the steps i missed?

[manuelbuil]

but when i start the rke2-server it never joins to the rest, do you see any error in the journalctl logs?

[rogersollberger]

this is the log i got from the server who doesnt want to join:

Jun 11 19:40:18 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:18Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:23 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:23Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:28 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:40:28.042656Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:40:28 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:28Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:40:28 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:28Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:33 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:33Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:38 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:38Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:43 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:40:43.042858Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:40:43 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:43Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:40:43 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:43Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:47 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:47Z" level=info msg="Waiting for etcd server to become available"
Jun 11 19:40:47 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:47Z" level=info msg="Waiting for API server to become available"
Jun 11 19:40:48 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:48Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:52 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:40:52.492149Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: i/o timeout\""}
Jun 11 19:40:52 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:52Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 11 19:40:53 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:53Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:40:58 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:40:58.043047Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:40:58 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:58Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:40:58 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:40:58Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:03 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:03Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:08 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:08Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:13 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:41:13.043221Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:41:13 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:13Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:41:13 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:13Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:17 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:17Z" level=info msg="Waiting for etcd server to become available"
Jun 11 19:41:17 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:17Z" level=info msg="Waiting for API server to become available"
Jun 11 19:41:18 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:18Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:23 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:23Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:27 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:41:27.492375Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: i/o timeout\""}
Jun 11 19:41:27 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:27Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 11 19:41:28 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:41:28.043418Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:41:28 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:28Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:41:28 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:28Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:33 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:33Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:38 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:38Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:43 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:41:43.043584Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:41:43 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:43Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:41:43 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:43Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:47 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:47Z" level=info msg="Waiting for etcd server to become available"
Jun 11 19:41:47 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:47Z" level=info msg="Waiting for API server to become available"
Jun 11 19:41:48 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:48Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:53 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:53Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:41:58 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:41:58.043762Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 11 19:41:58 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:58Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 11 19:41:58 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:41:58Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 11 19:42:02 tstedgelxm02neu rke2[191500]: {"level":"warn","ts":"2024-06-11T19:42:02.492606Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001040000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: i/o timeout\""}
Jun 11 19:42:02 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:42:02Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 11 19:42:03 tstedgelxm02neu rke2[191500]: time="2024-06-11T19:42:03Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"

[josh-at-straker]

Have a check in the etcd logs.

Last time I experienced this I had to forcefully remove the node from etcd, then join it back to the cluster. Master nodes can be a bit finicky, especially if the node name is the same which is often the case with VMs

Here's my note from the last time it occurred:

---

Use something like lens to get a shell on etcd pod... or something like
kubectl -n kube-system exec -it etcd-dev-k8s-alpha-failover-m-01 -- bash

Then run the following list all nodes:

etcdctl --cert /mnt/data/apps/kubernetes/server/tls/etcd/server-client.crt --key /mnt/data/apps/kubernetes/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /mnt/data/apps/kubernetes/server/tls/etcd/server-ca.crt member list

Then remove the broken member.

etcdctl --cert /mnt/data/kubernetes/server/tls/etcd/server-client.crt --key /mnt/data/kubernetes/server/tls/etcd/server-client.key --endpoints https://127.0.0.1:2379 --cacert /mnt/data/kubernetes/server/tls/etcd/server-ca.crt member remove <node-id from previous cmd>

[rogersollberger]

i tried out first following scenario: i deployed a master00 and worker00. I set them up and all is working fine.

Now when i try to add a second master, it never starts up. So in that it can't have this node already in the etcd list.
For me it looks strange, that after the first setup, everything works fine but then im not even able to add an master to the system.
workers just work fine when i add them. So i think i miss something. Because i think this should work normally.

the error log look like this:
un 12 09:59:17 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:17Z" level=info msg="Waiting for etcd server to become available"
Jun 12 09:59:21 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:21Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:21 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:21.789659Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 09:59:21 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:21Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 12 09:59:26 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:26Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:30 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:30.80955Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 09:59:30 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:30Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 09:59:31 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:31Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:31 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:31.821838Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 09:59:31 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T09:59:31.822878Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 09:59:36 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:36Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:41 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:41Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:45 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:45.81067Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 09:59:45 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:45Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 09:59:46 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:46Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:46 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:46.824974Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 09:59:46 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T09:59:46.825063Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 09:59:47 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:47Z" level=info msg="Waiting for etcd server to become available"
Jun 12 09:59:51 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:51Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:56 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:56Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 09:59:56 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T09:59:56.789979Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 09:59:56 tstedgelxm01neu rke2[27107]: time="2024-06-12T09:59:56Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 12 10:00:00 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:00.810915Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 10:00:00 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:00Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 10:00:01 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:01Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:01 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:01.826231Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 10:00:01 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T10:00:01.826339Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 10:00:06 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:06Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:11 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:11Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:15 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:15.811458Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 10:00:15 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:15Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 10:00:16 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:16Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:16 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:16.826934Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 10:00:16 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T10:00:16.827036Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 10:00:17 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:17Z" level=info msg="Waiting for etcd server to become available"
Jun 12 10:00:21 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:21Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:26 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:26Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:30 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:30.81217Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 10:00:30 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:30Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 10:00:31 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:31Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:31 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:31.791011Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 10:00:31 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:31Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jun 12 10:00:31 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:31.827702Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 10:00:31 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T10:00:31.82781Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 10:00:36 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:36Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:41 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:41Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:45 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:45.812671Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jun 12 10:00:45 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:45Z" level=error msg="Failed to check local etcd status for learner management: context deadline exceeded"
Jun 12 10:00:46 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:46Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jun 12 10:00:46 tstedgelxm01neu rke2[27107]: {"level":"warn","ts":"2024-06-12T10:00:46.828277Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000b788c0/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = "transport: authentication handshake failed: context deadline exceeded""}
Jun 12 10:00:46 tstedgelxm01neu rke2[27107]: {"level":"info","ts":"2024-06-12T10:00:46.828352Z","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 12 10:00:47 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:47Z" level=info msg="Waiting for etcd server to become available"
Jun 12 10:00:51 tstedgelxm01neu rke2[27107]: time="2024-06-12T10:00:51Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"

[rogersollberger]

here is the config.yaml of master 01

server: https://192.0.0.7:9345
token: XXXXXX

data-dir: /var/lib/rancher/rke2
cni: cilium
tls-san:
  - cluster.local
  - tstedgelxm01neu
  - 192.0.0.8
  - 192.0.0.254
node-taint:
  - CriticalAddonsOnly=true:NoExecute
node-label:
  - node_type=server
disable: ['rke2-ingress-nginx']
snapshotter: overlayfs
node-name: tstedgelxm01neu
node-ip: 192.0.0.8
disable-kube-proxy: true
write-kubeconfig-mode: "0644"
debug: false

and the az_ccm config (addition)

# master (node_type: server)
kube-controller-manager-arg:
  - "--cloud-provider=external"
disable-cloud-controller: true
kubelet-arg:
  - "--cloud-provider=external"