HELP Please 🥹 Cluster broke down due to namespace removal

[linus365]

I am glad for any help, I know I really messed up 😥

I don't know how to continue and the services and the data on the cluster are important.

Environmental Info:
RKE2 Version:
rke2 version v1.26.10+rke2r1 (9b01c8e)
go version go1.20.10 X:boringcrypto

Node(s) CPU architecture, OS, and Version:

3 nodes, all the same:
Linux node-01 5.4.0-166-generic #183-Ubuntu SMP Mon Oct 2 11:28:33 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration:
3 servers running as worker nodes as well as the control plane

Describe the bug:
I accidentally removed the namespace "local" or something like that and the cluster went offline and I can't fix it.
Tried to do a rke2 server --cluster-reset on one of the nodes but to no avail.

Steps To Reproduce:
Remove the local namespace on an rke2 cluster.

Expected behavior:
This should not simply destroy the cluster.

Actual behavior:
Destroyed the cluster.

Additional context / logs:

~ » sudo rke2 server
WARN[0000] not running in CIS mode                      
INFO[0000] Applying Pod Security Admission Configuration 
INFO[0000] Starting rke2 v1.26.10+rke2r1 (9b01c8e4e9f781a2654075f43cb59b922fed1713) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-11-15T00:07:26.16401+0100","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2400"]}
{"level":"info","ts":"2023-11-15T00:07:26.164209+0100","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]}
{"level":"info","ts":"2023-11-15T00:07:26.164304+0100","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.20.10 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":10,"max-cpu-available":10,"member-initialized":true,"name":"node-01-5ad0427b","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":0,"max-snapshots":0,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":["http://0.0.0.0:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-11-15T00:07:26.212098+0100","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"47.631117ms"}
{"level":"info","ts":"2023-11-15T00:07:26.962139+0100","caller":"etcdserver/server.go:509","msg":"recovered v2 store from snapshot","snapshot-index":727433169,"snapshot-size":"127 kB"}
{"level":"info","ts":"2023-11-15T00:07:26.962263+0100","caller":"etcdserver/server.go:522","msg":"recovered v3 backend from snapshot","backend-size-bytes":196882432,"backend-size":"197 MB","backend-size-in-use-bytes":56061952,"backend-size-in-use":"56 MB"}
{"level":"info","ts":"2023-11-15T00:07:27.208918+0100","caller":"etcdserver/raft.go:603","msg":"forcing restart member","cluster-id":"e38e60759b675315","local-member-id":"362ffb0f40a7f0a2","commit-index":727438380}
{"level":"info","ts":"2023-11-15T00:07:27.209285+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 switched to configuration voters=(3904615444881338530)"}
{"level":"info","ts":"2023-11-15T00:07:27.209348+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 became follower at term 6541"}
{"level":"info","ts":"2023-11-15T00:07:27.209376+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 362ffb0f40a7f0a2 [peers: [362ffb0f40a7f0a2], term: 6541, commit: 727438380, applied: 727433169, lastindex: 727438380, lastterm: 6541]"}
{"level":"info","ts":"2023-11-15T00:07:27.209605+0100","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-11-15T00:07:27.209642+0100","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"e38e60759b675315","local-member-id":"362ffb0f40a7f0a2","recovered-remote-peer-id":"362ffb0f40a7f0a2","recovered-remote-peer-urls":["https://<ip>:2380"]}
{"level":"info","ts":"2023-11-15T00:07:27.209705+0100","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-11-15T00:07:27.210834+0100","caller":"auth/store.go:1238","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-11-15T00:07:27.211754+0100","caller":"mvcc/kvstore.go:323","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":664910563}
{"level":"info","ts":"2023-11-15T00:07:27.269451+0100","caller":"mvcc/kvstore.go:393","msg":"kvstore restored","current-rev":664913465}
{"level":"info","ts":"2023-11-15T00:07:27.270814+0100","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-11-15T00:07:27.272084+0100","caller":"etcdserver/corrupt.go:95","msg":"starting initial corruption check","local-member-id":"362ffb0f40a7f0a2","timeout":"15s"}
{"level":"info","ts":"2023-11-15T00:07:27.292731+0100","caller":"etcdserver/corrupt.go:165","msg":"initial corruption checking passed; no corruption","local-member-id":"362ffb0f40a7f0a2"}
{"level":"info","ts":"2023-11-15T00:07:27.292904+0100","caller":"etcdserver/server.go:845","msg":"starting etcd server","local-member-id":"362ffb0f40a7f0a2","local-server-version":"3.5.9","cluster-id":"e38e60759b675315","cluster-version":"3.5"}
{"level":"info","ts":"2023-11-15T00:07:27.293272+0100","caller":"etcdserver/server.go:738","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"362ffb0f40a7f0a2","forward-ticks":9,"forward-duration":"4.5s","election-ticks":10,"election-timeout":"5s"}
{"level":"info","ts":"2023-11-15T00:07:27.297446+0100","caller":"embed/etcd.go:597","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-11-15T00:07:27.297593+0100","caller":"embed/etcd.go:569","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-11-15T00:07:27.297815+0100","caller":"embed/etcd.go:278","msg":"now serving peer/client/metrics","local-member-id":"362ffb0f40a7f0a2","initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":["http://0.0.0.0:2381"]}
{"level":"info","ts":"2023-11-15T00:07:27.297911+0100","caller":"embed/etcd.go:855","msg":"serving metrics","address":"http://0.0.0.0:2381"}
{"level":"info","ts":"2023-11-15T00:07:28.710987+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 is starting a new election at term 6541"}
{"level":"info","ts":"2023-11-15T00:07:28.711092+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 became pre-candidate at term 6541"}
{"level":"info","ts":"2023-11-15T00:07:28.711215+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 received MsgPreVoteResp from 362ffb0f40a7f0a2 at term 6541"}
{"level":"info","ts":"2023-11-15T00:07:28.711253+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 became candidate at term 6542"}
{"level":"info","ts":"2023-11-15T00:07:28.711271+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 received MsgVoteResp from 362ffb0f40a7f0a2 at term 6542"}
{"level":"info","ts":"2023-11-15T00:07:28.711292+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"362ffb0f40a7f0a2 became leader at term 6542"}
{"level":"info","ts":"2023-11-15T00:07:28.711312+0100","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 362ffb0f40a7f0a2 elected leader 362ffb0f40a7f0a2 at term 6542"}
{"level":"info","ts":"2023-11-15T00:07:28.737022+0100","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-11-15T00:07:28.737122+0100","caller":"embed/serve.go:103","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-11-15T00:07:28.737013+0100","caller":"etcdserver/server.go:2064","msg":"published local member to cluster through raft","local-member-id":"362ffb0f40a7f0a2","local-member-attributes":"{Name:node-01-5ad0427b ClientURLs:[http://127.0.0.1:2399]}","request-path":"/0/members/362ffb0f40a7f0a2/attributes","cluster-id":"e38e60759b675315","publish-timeout":"15s"}
{"level":"info","ts":"2023-11-15T00:07:28.737712+0100","caller":"embed/serve.go:187","msg":"serving client traffic insecurely; this is strongly discouraged!","traffic":"http","address":"127.0.0.1:2402"}
{"level":"info","ts":"2023-11-15T00:07:28.739248+0100","caller":"embed/serve.go:187","msg":"serving client traffic insecurely; this is strongly discouraged!","traffic":"grpc","address":"127.0.0.1:2399"}
INFO[0003] Defragmenting etcd database                  
{"level":"info","ts":"2023-11-15T00:07:28.743449+0100","caller":"v3rpc/maintenance.go:90","msg":"starting defragment"}
{"level":"info","ts":"2023-11-15T00:07:28.746458+0100","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":196882432,"current-db-size":"197 MB","current-db-size-in-use-bytes":56066048,"current-db-size-in-use":"56 MB"}
{"level":"info","ts":"2023-11-15T00:07:29.50059+0100","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-141029376,"current-db-size-bytes":55853056,"current-db-size":"56 MB","current-db-size-in-use-bytes-diff":-221184,"current-db-size-in-use-bytes":55844864,"current-db-size-in-use":"56 MB","took":"757.023512ms"}
{"level":"info","ts":"2023-11-15T00:07:29.500676+0100","caller":"v3rpc/maintenance.go:96","msg":"finished defragment"}
{"level":"warn","ts":"2023-11-15T00:07:29.500706+0100","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-11-15T00:07:28.743374+0100","time spent":"757.325224ms","remote":"127.0.0.1:59222","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
INFO[0003] etcd temporary data store connection OK      
INFO[0003] Reconciling bootstrap data between datastore and disk 
INFO[0004] stopping etcd                                
{"level":"info","ts":"2023-11-15T00:07:29.53011+0100","caller":"embed/etcd.go:376","msg":"closing etcd server","name":"node-01-5ad0427b","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"]}
{"level":"info","ts":"2023-11-15T00:07:29.531195+0100","caller":"etcdserver/server.go:1465","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"362ffb0f40a7f0a2","current-leader-member-id":"362ffb0f40a7f0a2"}
INFO[0004] Starting etcd for existing cluster member    
INFO[0004] start                                        
INFO[0004] schedule, now=2023-11-15T00:07:29+01:00, entry=1, next=2023-11-15T12:00:00+01:00 
INFO[0004] certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1620936637: notBefore=2021-05-13 20:10:37 +0000 UTC notAfter=2024-11-13 23:07:29 +0000 UTC 
WARN[0004] dynamiclistener [::]:9345: no cached certificate available for preload - deferring certificate load until storage initialization or first client request 
INFO[0004] Active TLS secret / (ver=) (count 12): map[listener.cattle.io/cn-10.43.0.1:10.43.0.1 listener.cattle.io/cn-127.0.0.1:127.0.0.1 listener.cattle.io/cn<ip> listener.cattle.io/cn-__1-f16284:::1 listener.cattle.io/cn-k3s.365my.biz:k3s.365my.biz listener.cattle.io/cn-kubernetes:kubernetes listener.cattle.io/cn-kubernetes.default:kubernetes.default listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local listener.cattle.io/cn-localhost:localhost listener.cattle.io/cn-node-01:node-01 listener.cattle.io/cn-node-01.365my.biz:node-01.365my.biz listener.cattle.io/fingerprint:SHA1=B1AD36D0EDD95756A55BA9D251FEA934EEC4691B] 
{"level":"info","ts":"2023-11-15T00:07:29.569439+0100","caller":"embed/etcd.go:579","msg":"stopping serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-11-15T00:07:29.569883+0100","caller":"embed/etcd.go:584","msg":"stopped serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-11-15T00:07:29.570021+0100","caller":"embed/etcd.go:378","msg":"closed etcd server","name":"node-01-5ad0427b","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"]}
INFO[0004] Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --authorization-mode=Node,RBAC --bind-address=0.0.0.0 --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml --enable-admission-plugins=NodeRestriction --enable-aggregator-routing=true --enable-bootstrap-token-auth=true --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key --etcd-servers=https://127.0.0.1:2379 --feature-gates=JobTrackingWithFinalizers=true --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --profiling=false --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key --requestheader-allowed-names=system:auth-proxy --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt --requestheader-extra-headers-prefix=X-Remote-Extra- --requestheader-group-headers=X-Remote-Group --requestheader-username-headers=X-Remote-User --secure-port=6443 --service-account-issuer=https://kubernetes.default.svc.cluster.local --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --service-node-port-range=30000-32767 --storage-backend=etcd3 --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt --tls-cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305 --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key 
INFO[0004] Removed kube-apiserver static pod manifest   
INFO[0004] Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --bind-address=127.0.0.1 --kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --profiling=false --secure-port=10259 
INFO[0004] Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.42.0.0/16 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.nochain.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.nochain.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --configure-cloud-routes=false --controllers=*,tokencleaner,-service,-route,-cloud-node-lifecycle --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.current.key --service-cluster-ip-range=10.43.0.0/16 --use-service-account-credentials=true 
INFO[0004] Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --bind-address=127.0.0.1 --cloud-config=/var/lib/rancher/rke2/server/etc/cloud-config.yaml --cloud-provider=rke2 --cluster-cidr=10.42.0.0/16 --configure-cloud-routes=false --controllers=*,-route,-service --kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --leader-elect-resource-name=rke2-cloud-controller-manager --node-status-update-frequency=1m0s --profiling=false 
INFO[0004] Server node token is available at /var/lib/rancher/rke2/server/token 
INFO[0004] Waiting for cri connection: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused" 
INFO[0004] To join server node to cluster: rke2 server -s https://<ip>:9345 -t ${SERVER_NODE_TOKEN} 
INFO[0004] Agent node token is available at /var/lib/rancher/rke2/server/agent-token 
INFO[0004] To join agent node to cluster: rke2 agent -s https:/<ip>:9345 -t ${AGENT_NODE_TOKEN} 
INFO[0004] Wrote kubeconfig /etc/rancher/rke2/rke2.yaml 
INFO[0004] Run: rke2 kubectl                            
INFO[0004] Handling backend connection request [node-02] 
INFO[0005] Password verified locally for node node-01   
INFO[0005] certificate CN=node-01 signed by CN=rke2-server-ca@1620936637: notBefore=2021-05-13 20:10:37 +0000 UTC notAfter=2024-11-13 23:07:30 +0000 UTC 
INFO[0005] certificate CN=system:node:node-01,O=system:nodes signed by CN=rke2-client-ca@1620936637: notBefore=2021-05-13 20:10:37 +0000 UTC notAfter=2024-11-13 23:07:31 +0000 UTC 
INFO[0005] Module overlay was already loaded            
INFO[0005] Module nf_conntrack was already loaded       
INFO[0005] Module br_netfilter was already loaded       
INFO[0005] Module iptable_nat was already loaded        
INFO[0005] Module iptable_filter was already loaded     
INFO[0005] Runtime image index.docker.io/rancher/rke2-runtime:v1.26.10-rke2r1 bin and charts directories already exist; skipping extract 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-validation-webhook.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller-crd.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values 
INFO[0005] Updated manifest /var/lib/rancher/rke2/server/manifests/rke2-snapshot-controller.yaml to set cluster configuration values 
INFO[0005] Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log 
INFO[0005] Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd 
INFO[0006] Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused" 
INFO[0007] Waiting for containerd startup: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /run/k3s/containerd/containerd.sock: connect: connection refused" 
INFO[0008] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet 
INFO[0009] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet 
INFO[0010] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet 
INFO[0011] Waiting for containerd startup: rpc error: code = Unknown desc = server is not initialized yet 
INFO[0012] containerd is now running                    
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 1.481925ms 
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 789.218µs 
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 850.445µs 
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 789.779µs 
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt in 417.213µs 
INFO[0013] Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt 
INFO[0013] Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 790.261µs 
INFO[0013] Connecting to proxy                           url="wss://127.0.0.1:9345/v1-rke2/connect"
INFO[0013] Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=cgroupfs --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=node-01 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --kubelet-cgroups=/rke2 --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=<ip> --node-labels=server=node-01 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key 
INFO[0013] Handling backend connection request [node-01] 
INFO[0013] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
INFO[0018] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 
{"level":"warn","ts":"2023-11-15T00:07:44.534465+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0002cd500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
{"level":"info","ts":"2023-11-15T00:07:44.534705+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
INFO[0020] Defragmenting etcd database                  
INFO[0021] etcd data store connection OK                
INFO[0021] Saving cluster bootstrap data to datastore   
INFO[0021] Waiting for API server to become available   
WARN[0021] Bootstrap key already exists                 
INFO[0023] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error 

[brandond]

I'm not sure what you mean by Remove the local namespace on an rke2 cluster. There is no "local" namespace by default; what exactly did you delete? If you look at the kube-apiserver pod logs, what is preventing it from becoming ready?

Tried to do a rke2 server --cluster-reset on one of the nodes but to no avail.

That just resets the etcd cluster to a single node; did you want to restore from a snapshot? Do you have any etcd snapshots available to restore from?

[linus365]

The problem is, i used k9s and must have selected the wrong namespace for termination. I think it was called something like local.
I rebooted the node i reset before and it came back up:

node-01   Ready    control-plane,etcd,master   592d    v1.26.10+rke2r1

Now I see these logs in journalctl for rke2-server

Nov 15 00:40:53 node-01 rke2[974]: time="2023-11-15T00:40:53+01:00" level=info msg="Removing etcd member from cluster due to node delete" address=<ip> name=node-01-5ad0427b
Nov 15 00:40:53 node-01 rke2[974]: time="2023-11-15T00:40:53+01:00" level=info msg="Removing name=node-01-5ad0427b id=3904615444881338530 address=<ip> from etcd"
Nov 15 00:40:53 node-01 rke2[974]: {"level":"warn","ts":"2023-11-15T00:40:53.524091+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000735500/127.0.0.1:2379","attempt":0,"error":"rpc error: code = Unknown desc = etcdserver: re-configuration failed due to not enough started members"}
Nov 15 00:40:53 node-01 rke2[974]: time="2023-11-15T00:40:53+01:00" level=error msg="error syncing 'node-01': handler managed-etcd-controller: etcdserver: re-configuration failed due to not enough started members, requeuing"

[brandond]

Removing etcd member from cluster due to node delete" address=<ip> name=node-01-5ad0427b

It appears that you also deleted the node objects from Kubernetes? That is going to be somewhat harder to recover from; you might have to clear the finalizers on the last node in the cluster and allow it to finish deleting, at which point you can restart again and allow the node to be recreated.

[linus365]

I restarted rke2-server on another node and that has these logs now:

~ » sudo journalctl -fu rke2-server.service
-- Logs begin at Sat 2023-09-02 03:14:30 CEST. --
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-proxy-image.txt in 718.788µs"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 388.559µs"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Running kubelet --address=0.0.0.0 --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=node-03 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --log-file=/var/lib/rancher/rke2/agent/logs/kubelet.log --log-file-max-size=50 --logtostderr=false --node-ip=<ip> --node-labels=server=node-03 --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/run/systemd/resolve/resolv.conf --serialize-image-pulls=false --stderrthreshold=FATAL --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Connecting to proxy" url="wss://127.0.0.1:9345/v1-rke2/connect"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Handling backend connection request [node-03]"
Nov 15 00:42:48 node-03 rke2[957]: time="2023-11-15T00:42:48+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Nov 15 00:42:50 node-03 rke2[957]: {"level":"warn","ts":"2023-11-15T00:42:50.841798+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d62000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
Nov 15 00:42:50 node-03 rke2[957]: {"level":"info","ts":"2023-11-15T00:42:50.842147+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Nov 15 00:42:53 node-03 rke2[957]: time="2023-11-15T00:42:53+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Nov 15 00:42:56 node-03 rke2[957]: time="2023-11-15T00:42:56+01:00" level=info msg="Pod for etcd not synced (waiting for termination of old pod sandbox), retrying"
Nov 15 00:42:58 node-03 rke2[957]: time="2023-11-15T00:42:58+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Nov 15 00:43:03 node-03 rke2[957]: time="2023-11-15T00:43:03+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
:qNov 15 00:43:04 node-03 rke2[957]: {"level":"warn","ts":"2023-11-15T00:43:04.577556+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d62000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
Nov 15 00:43:04 node-03 rke2[957]: {"level":"info","ts":"2023-11-15T00:43:04.577694+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Nov 15 00:43:04 node-03 rke2[957]: {"level":"warn","ts":"2023-11-15T00:43:04.588848+0100","logger":"etcd-client","caller":"v3@v3.5.9-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d62000/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = latest balancer error: last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:2379: connect: connection refused\""}
Nov 15 00:43:04 node-03 rke2[957]: time="2023-11-15T00:43:04+01:00" level=info msg="Failed to test data store connection: context deadline exceeded"
Nov 15 00:43:04 node-03 rke2[957]: time="2023-11-15T00:43:04+01:00" level=info msg="Waiting for etcd server to become available"
Nov 15 00:43:04 node-03 rke2[957]: time="2023-11-15T00:43:04+01:00" level=info msg="Waiting for API server to become available"
Nov 15 00:43:07 node-03 rke2[957]: time="2023-11-15T00:43:07+01:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"

It seems it's not coming up

[linus365]

Let me try that

[linus365]

INFO[0022] certificate CN=etcd-peer signed by CN=etcd-peer-ca@1620936637: notBefore=2021-05-13 20:10:37 +0000 UTC notAfter=2024-11-14 00:23:26 +0000 UTC 
INFO[0022] certificate CN=etcd-server signed by CN=etcd-server-ca@1620936637: notBefore=2021-05-13 20:10:37 +0000 UTC notAfter=2024-11-14 00:23:26 +0000 UTC 
INFO[0022] Shutting down kubelet and etcd               
ERRO[0022] Kubelet exited: signal: killed               
INFO[0026] Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:6444/v1-rke2/readyz: 500 Internal Server Error 
INFO[0027] Managed etcd cluster membership has been reset, restart without --cluster-reset flag now. Backup and delete ${datadir}/server/db on each peer etcd server and rejoin the nodes

yes, thank you

[linus365]

You are my absolute hero, thank you so much. All nodes are back:

→ kgn                                                                                                                                                                                                                          
NAME      STATUS   ROLES                       AGE    VERSION
node-01   Ready    control-plane,etcd,master   592d   v1.26.10+rke2r1
node-02   Ready    control-plane,etcd,master   603d   v1.26.10+rke2r1
node-03   Ready    control-plane,etcd,master   603d   v1.26.10+rke2r1

[linus365]

This was the namespace I removed:

→ kgns local -o yaml                                                                                                                                                                                                            able 
apiVersion: v1
kind: Namespace
metadata:
  annotations:
    cattle.io/status: '{"Conditions":[{"Type":"ResourceQuotaInit","Status":"True","Message":"","LastUpdateTime":"2021-08-08T19:40:55Z"},{"Type":"InitialRolesPopulated","Status":"True","Message":"","LastUpdateTime":"2021-08-08T19:40:57Z"}]}'
    lifecycle.cattle.io/create.namespace-auth: "true"
    management.cattle.io/no-default-sa-token: "true"
    management.cattle.io/system-namespace: "true"
  creationTimestamp: "2021-08-08T19:40:32Z"
  finalizers:
  - controller.cattle.io/namespace-auth
  labels:
    kubernetes.io/metadata.name: local
  name: local
  resourceVersion: "124746963"
  uid: b8d826bc-e68f-4785-bec0-40dc36f1b5a2
spec:
  finalizers:
  - kubernetes
status:
  phase: Active

[linus365]

It's all stabilized now, fixing the last longhorn volume by re-importing the backups I have.
Thank you so much Brandon for your help.
I have no idea yet what happened to cause this and will have to dig into this to avoid re-occurrences in the future.
Thanks 🙇🏼